[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:02:31 dist-test-slave-zpbw krb5kdc[24996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:02:31 dist-test-slave-zpbw krb5kdc[24996](info): set up 2 sockets
Jun 19 15:02:31 dist-test-slave-zpbw krb5kdc[24996](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 15:02:32.343300 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:02:32 dist-test-slave-zpbw krb5kdc[24996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:02:32.519181 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:33747
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:38711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:33747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:32.849714 25009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:32.850030 25009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:32.850121 25009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:32.857676 25009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:02:32.857796 25009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:32.857936 25009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:32.858034 25009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:02:32.858116 25009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:02:32.868520 25009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:33747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:33747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:32.869966 25009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:32.871528 25009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:32.886183 25014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:32.886209 25015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:32.887279 25017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:32.888386 25009 server_base.cc:1034] running on GCE node
I20260619 15:02:32.889570 25009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:32.891446 25009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:32.892684 25009 hybrid_clock.cc:648] HybridClock initialized: now 1781881352892636 us; error 45 us; skew 500 ppm
I20260619 15:02:32.893044 25009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:32 dist-test-slave-zpbw krb5kdc[24996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881352, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:32.900732 25009 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:02:32.902382 25009 webserver.cc:460] Webserver started at http://127.24.103.126:41423/ using document root <none> and password file <none>
I20260619 15:02:32.903199 25009 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:32.903343 25009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:32.903674 25009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:32.906836 25009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "b192076b36864376a62d5c0b1db68e89"
format_stamp: "Formatted at 2026-06-19 15:02:32 on dist-test-slave-zpbw"
server_key: "8d3fef4a83cbb2885a5d8b6ef6540dcc"
server_key_iv: "73905e5d72532f09da716dea7f81187c"
server_key_version: "encryptionkey@0"
I20260619 15:02:32.907652 25009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "b192076b36864376a62d5c0b1db68e89"
format_stamp: "Formatted at 2026-06-19 15:02:32 on dist-test-slave-zpbw"
server_key: "8d3fef4a83cbb2885a5d8b6ef6540dcc"
server_key_iv: "73905e5d72532f09da716dea7f81187c"
server_key_version: "encryptionkey@0"
I20260619 15:02:32.913180 25009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 15:02:32.916927 25024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:32.918352 25009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 15:02:32.918556 25009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "b192076b36864376a62d5c0b1db68e89"
format_stamp: "Formatted at 2026-06-19 15:02:32 on dist-test-slave-zpbw"
server_key: "8d3fef4a83cbb2885a5d8b6ef6540dcc"
server_key_iv: "73905e5d72532f09da716dea7f81187c"
server_key_version: "encryptionkey@0"
I20260619 15:02:32.918773 25009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:32.945595 25009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:32.949812 25009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:32.950124 25009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:32.969460 25009 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:33747
I20260619 15:02:32.969497 25075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:33747 every 8 connection(s)
I20260619 15:02:32.970989 25009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:02:32.974823 25076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:32.977360 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 25009
I20260619 15:02:32.977689 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:02:32.980149 24989 external_mini_cluster.cc:1428] Setting key a715c560a9e198a27077a144dc7e27e6
I20260619 15:02:32.989120 25076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89: Bootstrap starting.
I20260619 15:02:32.992345 25076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:32.993423 25076 log.cc:824] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:32.996675 25076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89: No bootstrap required, opened a new log
I20260619 15:02:33.004410 25076 raft_consensus.cc:348] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b192076b36864376a62d5c0b1db68e89" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33747 } }
I20260619 15:02:33.004873 25076 raft_consensus.cc:374] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:33.005012 25076 raft_consensus.cc:729] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b192076b36864376a62d5c0b1db68e89, State: Initialized, Role: FOLLOWER
I20260619 15:02:33.005632 25076 consensus_queue.cc:260] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b192076b36864376a62d5c0b1db68e89" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33747 } }
I20260619 15:02:33.005908 25076 raft_consensus.cc:388] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:02:33.006050 25076 raft_consensus.cc:482] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:02:33.006227 25076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:33.008520 25076 raft_consensus.cc:504] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b192076b36864376a62d5c0b1db68e89" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33747 } }
I20260619 15:02:33.009063 25076 leader_election.cc:302] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b192076b36864376a62d5c0b1db68e89; no voters:
Jun 19 15:02:33 dist-test-slave-zpbw krb5kdc[24996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:33.009688 25076 leader_election.cc:288] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:02:33.009857 25081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:33.011121 25081 raft_consensus.cc:686] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [term 1 LEADER]: Becoming Leader. State: Replica: b192076b36864376a62d5c0b1db68e89, State: Running, Role: LEADER
I20260619 15:02:33.011759 25081 consensus_queue.cc:237] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b192076b36864376a62d5c0b1db68e89" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33747 } }
I20260619 15:02:33.012215 25076 sys_catalog.cc:564] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:02:33.015141 25082 sys_catalog.cc:455] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b192076b36864376a62d5c0b1db68e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b192076b36864376a62d5c0b1db68e89" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33747 } } }
I20260619 15:02:33.015543 25082 sys_catalog.cc:458] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [sys.catalog]: This master's current role is: LEADER
I20260619 15:02:33.016574 25083 sys_catalog.cc:455] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b192076b36864376a62d5c0b1db68e89. Latest consensus state: current_term: 1 leader_uuid: "b192076b36864376a62d5c0b1db68e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b192076b36864376a62d5c0b1db68e89" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33747 } } }
I20260619 15:02:33.019738 25083 sys_catalog.cc:458] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89 [sys.catalog]: This master's current role is: LEADER
I20260619 15:02:33.021395 25087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:02:33.027364 25087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:02:33.029667 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:32.985013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.24.103.126:33747)
0619 15:02:32.987251 (+ 2238us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:32.987267 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:32.987323 (+ 56us) server_negotiation.cc:407] Connection header received
0619 15:02:32.996211 (+ 8888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:32.996236 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:32.996644 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:32.997124 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:33.000310 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:33.001787 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:33.003718 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:33.004475 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:33.004704 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:33.011803 (+ 7099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:33.011816 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:33.011837 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:33.011899 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:33.020128 (+ 8229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:33.021757 (+ 1629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:33.021766 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:33.021774 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:33.022006 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:33.022725 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:33.022730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:33.022733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:33.023749 (+ 1016us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:33.027049 (+ 3300us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:33.027490 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 15:02:33.027888 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1949,"thread_start_us":1679,"threads_started":1}
I20260619 15:02:33.040405 25087 catalog_manager.cc:1347] Generated new cluster ID: d3873f85844843b5850367c66bc0bea5
I20260619 15:02:33.040632 25087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:02:33.069857 25087 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:02:33.071878 25087 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:02:33.097173 25087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b192076b36864376a62d5c0b1db68e89: Generated new TSK 0
I20260619 15:02:33.098336 25087 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:02:33.150101 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33747
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:38711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:33.497831 25102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:33.498128 25102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:33.498240 25102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:33.505859 25102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:33.505986 25102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:33.506199 25102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:02:33.516664 25102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:33.517966 25102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:33.519359 25102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:33.533416 25107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:33.533511 25108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:33.533442 25110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:33.538823 25102 server_base.cc:1034] running on GCE node
I20260619 15:02:33.539579 25102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:33.540583 25102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:33.541791 25102 hybrid_clock.cc:648] HybridClock initialized: now 1781881353541730 us; error 60 us; skew 500 ppm
I20260619 15:02:33.542122 25102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:33 dist-test-slave-zpbw krb5kdc[24996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881353, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:33.545619 25102 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:02:33.547449 25102 webserver.cc:460] Webserver started at http://127.24.103.65:33757/ using document root <none> and password file <none>
I20260619 15:02:33.548192 25102 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:33.548326 25102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:33.548656 25102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:33.551950 25102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "344ef81b544a49438b2276e0c2016071"
format_stamp: "Formatted at 2026-06-19 15:02:33 on dist-test-slave-zpbw"
server_key: "0a6dab073fe3246e25d83137a96774d0"
server_key_iv: "69bc36465e627f605cf281a2abf384ad"
server_key_version: "encryptionkey@0"
I20260619 15:02:33.552785 25102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "344ef81b544a49438b2276e0c2016071"
format_stamp: "Formatted at 2026-06-19 15:02:33 on dist-test-slave-zpbw"
server_key: "0a6dab073fe3246e25d83137a96774d0"
server_key_iv: "69bc36465e627f605cf281a2abf384ad"
server_key_version: "encryptionkey@0"
I20260619 15:02:33.558346 25102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:02:33.562099 25117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:33.563604 25102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 15:02:33.563802 25102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "344ef81b544a49438b2276e0c2016071"
format_stamp: "Formatted at 2026-06-19 15:02:33 on dist-test-slave-zpbw"
server_key: "0a6dab073fe3246e25d83137a96774d0"
server_key_iv: "69bc36465e627f605cf281a2abf384ad"
server_key_version: "encryptionkey@0"
I20260619 15:02:33.564035 25102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:33.582197 25102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:33.586551 25102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:33.586939 25102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:33.587957 25102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:02:33.589406 25102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:33.589515 25102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:33.589671 25102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:33.589751 25102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:33.626416 25102 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:38483
I20260619 15:02:33.626454 25229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:38483 every 8 connection(s)
I20260619 15:02:33.627889 25102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:02:33.629437 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 25102
I20260619 15:02:33.629734 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:02:33.630514 24989 external_mini_cluster.cc:1428] Setting key 2047812d15c90e440ff21b1d834d5efa
Jun 19 15:02:33 dist-test-slave-zpbw krb5kdc[24996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881353, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
I20260619 15:02:33.653520 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:33.631179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:39061 (local address 127.24.103.126:33747)
0619 15:02:33.631718 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:33.631725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:33.633025 (+ 1300us) server_negotiation.cc:407] Connection header received
0619 15:02:33.636313 (+ 3288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:33.636317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:33.636398 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:33.636498 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:33.638676 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:33.639936 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:33.641603 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:33.642348 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:33.642499 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:33.646647 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:33.646656 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:33.646662 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:33.646714 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:33.649930 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:33.651038 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:33.651042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:33.651044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:33.651168 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:33.651667 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:33.651672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:33.651675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:33.651954 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:33.652112 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:33.652877 (+ 765us) server_negotiation.cc:299] Negotiation successful
0619 15:02:33.653116 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":265,"threads_started":1}
I20260619 15:02:33.654573 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:33.631894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33747 (local address 127.24.103.65:39061)
0619 15:02:33.632859 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:33.632910 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:33.635946 (+ 3036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:33.636825 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:33.636841 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:33.637714 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:33.638474 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:33.638492 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:33.640128 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:33.640133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:33.641437 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:33.641449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:33.642583 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:33.642589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:33.642959 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:33.643814 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:33.643848 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:33.646341 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:33.650116 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:33.650125 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:33.650140 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:33.650896 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:33.651308 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:33.651314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:33.651318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:33.651541 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:33.652089 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:33.652098 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:33.652476 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 15:02:33.652770 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 15:02:33.653356 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":366,"threads_started":1}
I20260619 15:02:33.656378 25230 heartbeater.cc:347] Connected to a master server at 127.24.103.126:33747
I20260619 15:02:33.656759 25230 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:33.657797 25230 heartbeater.cc:511] Master 127.24.103.126:33747 requested a full tablet report, sending...
I20260619 15:02:33.660331 25041 ts_manager.cc:194] Registered new tserver with Master: 344ef81b544a49438b2276e0c2016071 (127.24.103.65:38483)
I20260619 15:02:33.661999 25041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:39061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:02:33.673971 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33747
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:38711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:34.030102 25236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:34.030519 25236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:34.030632 25236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:34.038023 25236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:34.038157 25236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:34.038275 25236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:02:34.048751 25236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:34.050006 25236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:34.051432 25236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:34.065392 25244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:34.065485 25242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:34.065462 25241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:34.066372 25236 server_base.cc:1034] running on GCE node
I20260619 15:02:34.067128 25236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:34.068110 25236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:34.069319 25236 hybrid_clock.cc:648] HybridClock initialized: now 1781881354069241 us; error 74 us; skew 500 ppm
I20260619 15:02:34.069692 25236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:34 dist-test-slave-zpbw krb5kdc[24996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881354, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:34.073024 25236 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:02:34.074716 25236 webserver.cc:460] Webserver started at http://127.24.103.66:34515/ using document root <none> and password file <none>
I20260619 15:02:34.075490 25236 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:34.075651 25236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:34.075976 25236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:34.079164 25236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "7d2c6e61d933432c931afb49683fb81f"
format_stamp: "Formatted at 2026-06-19 15:02:34 on dist-test-slave-zpbw"
server_key: "f2717eb287fc2be83ad0fae421336d9e"
server_key_iv: "46cb49bd0e35b9946dce6893be8b67fc"
server_key_version: "encryptionkey@0"
I20260619 15:02:34.079977 25236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "7d2c6e61d933432c931afb49683fb81f"
format_stamp: "Formatted at 2026-06-19 15:02:34 on dist-test-slave-zpbw"
server_key: "f2717eb287fc2be83ad0fae421336d9e"
server_key_iv: "46cb49bd0e35b9946dce6893be8b67fc"
server_key_version: "encryptionkey@0"
I20260619 15:02:34.085747 25236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:02:34.089452 25251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:34.090958 25236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 15:02:34.091148 25236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "7d2c6e61d933432c931afb49683fb81f"
format_stamp: "Formatted at 2026-06-19 15:02:34 on dist-test-slave-zpbw"
server_key: "f2717eb287fc2be83ad0fae421336d9e"
server_key_iv: "46cb49bd0e35b9946dce6893be8b67fc"
server_key_version: "encryptionkey@0"
I20260619 15:02:34.091377 25236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:34.108532 25236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:34.112931 25236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:34.113268 25236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:34.114276 25236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:02:34.115890 25236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:34.115998 25236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:34.116149 25236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:34.116220 25236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:34.150393 25236 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:33951
I20260619 15:02:34.150430 25363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:33951 every 8 connection(s)
I20260619 15:02:34.151775 25236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:02:34.153123 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 25236
I20260619 15:02:34.153340 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:02:34.153880 24989 external_mini_cluster.cc:1428] Setting key d85b5498add601c210fad0ce0b1947b4
Jun 19 15:02:34 dist-test-slave-zpbw krb5kdc[24996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881354, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
I20260619 15:02:34.173964 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:34.154523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:36941 (local address 127.24.103.126:33747)
0619 15:02:34.155099 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:34.155106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:34.158147 (+ 3041us) server_negotiation.cc:407] Connection header received
0619 15:02:34.158204 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:34.158208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:34.158285 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:34.158411 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:34.160127 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.160855 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.162418 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.162905 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.163047 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:34.167397 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:34.167403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:34.167406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:34.167454 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:34.170476 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:34.171467 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:34.171472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:34.171474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:34.171599 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:34.172063 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:34.172067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:34.172069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:34.172344 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:34.172501 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:34.173309 (+ 808us) server_negotiation.cc:299] Negotiation successful
0619 15:02:34.173543 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":345,"threads_started":1}
I20260619 15:02:34.175173 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:34.155256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33747 (local address 127.24.103.66:36941)
0619 15:02:34.156074 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:34.156132 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:34.157845 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:34.158674 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:34.158683 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:34.159308 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:34.159939 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:34.159959 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.161019 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.161024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:34.162267 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:34.162279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.163141 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.163148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:34.163411 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:34.164149 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:34.164175 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:34.167117 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:34.170635 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:34.170646 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:34.170666 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:34.171320 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:34.171723 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:34.171728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:34.171732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:34.171933 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:34.172476 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:34.172484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:34.172898 (+ 414us) client_negotiation.cc:769] Sending connection context
0619 15:02:34.173203 (+ 305us) client_negotiation.cc:240] Negotiation successful
0619 15:02:34.173775 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":286,"threads_started":1}
I20260619 15:02:34.176779 25364 heartbeater.cc:347] Connected to a master server at 127.24.103.126:33747
I20260619 15:02:34.177101 25364 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:34.177852 25364 heartbeater.cc:511] Master 127.24.103.126:33747 requested a full tablet report, sending...
I20260619 15:02:34.179476 25041 ts_manager.cc:194] Registered new tserver with Master: 7d2c6e61d933432c931afb49683fb81f (127.24.103.66:33951)
I20260619 15:02:34.180817 25041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:36941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:02:34.195732 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33747
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:38711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:34.541363 25370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:34.541761 25370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:34.541940 25370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:34.549902 25370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:34.550041 25370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:34.550238 25370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:02:34.560734 25370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:34.562198 25370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:34.563606 25370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:34.579680 25376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:34.579612 25378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:34.579646 25375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:34.580291 25370 server_base.cc:1034] running on GCE node
I20260619 15:02:34.581015 25370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:34.582026 25370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:34.583267 25370 hybrid_clock.cc:648] HybridClock initialized: now 1781881354583193 us; error 61 us; skew 500 ppm
I20260619 15:02:34.583595 25370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:34 dist-test-slave-zpbw krb5kdc[24996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881354, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:34.586745 25370 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:02:34.588268 25370 webserver.cc:460] Webserver started at http://127.24.103.67:34217/ using document root <none> and password file <none>
I20260619 15:02:34.589032 25370 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:34.589186 25370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:34.589510 25370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:34.592519 25370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "888a0f18c6f64a5e8bea26b3469d2741"
format_stamp: "Formatted at 2026-06-19 15:02:34 on dist-test-slave-zpbw"
server_key: "d512c8ab908e7ce1b9a59bc9512a2f52"
server_key_iv: "3519d00d1c7541a426fd3b7b61d36f75"
server_key_version: "encryptionkey@0"
I20260619 15:02:34.593327 25370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "888a0f18c6f64a5e8bea26b3469d2741"
format_stamp: "Formatted at 2026-06-19 15:02:34 on dist-test-slave-zpbw"
server_key: "d512c8ab908e7ce1b9a59bc9512a2f52"
server_key_iv: "3519d00d1c7541a426fd3b7b61d36f75"
server_key_version: "encryptionkey@0"
I20260619 15:02:34.598986 25370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 15:02:34.602990 25385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:34.604504 25370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:02:34.604719 25370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "888a0f18c6f64a5e8bea26b3469d2741"
format_stamp: "Formatted at 2026-06-19 15:02:34 on dist-test-slave-zpbw"
server_key: "d512c8ab908e7ce1b9a59bc9512a2f52"
server_key_iv: "3519d00d1c7541a426fd3b7b61d36f75"
server_key_version: "encryptionkey@0"
I20260619 15:02:34.604929 25370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:34.645959 25370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:34.650300 25370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:34.650630 25370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:34.651686 25370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:02:34.653088 25370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:34.653195 25370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:34.653354 25370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:34.653426 25370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:34.666294 25230 heartbeater.cc:503] Master 127.24.103.126:33747 was elected leader, sending a full tablet report...
I20260619 15:02:34.691636 25370 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:33125
I20260619 15:02:34.691743 25497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:33125 every 8 connection(s)
I20260619 15:02:34.693001 25370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:02:34.695300 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 25370
I20260619 15:02:34.695534 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:02:34.696067 24989 external_mini_cluster.cc:1428] Setting key ff38e281baa456cb938fb1e37b000578
Jun 19 15:02:34 dist-test-slave-zpbw krb5kdc[24996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881354, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:34.714670 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:34.695756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:55487 (local address 127.24.103.126:33747)
0619 15:02:34.696319 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:34.696326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:34.697474 (+ 1148us) server_negotiation.cc:407] Connection header received
0619 15:02:34.700036 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:34.700042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:34.700138 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:34.700257 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:34.701831 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.702620 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.704339 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.704811 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.704955 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:34.708566 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:34.708571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:34.708574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:34.708609 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:34.711447 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:34.712387 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:34.712394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:34.712397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:34.712576 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:34.713047 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:34.713051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:34.713053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:34.713330 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:34.713442 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:34.714240 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 15:02:34.714405 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":253,"threads_started":1}
I20260619 15:02:34.715747 25501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:34.696430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33747 (local address 127.24.103.67:55487)
0619 15:02:34.697307 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:34.697359 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:34.699713 (+ 2354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:34.700512 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:34.700521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:34.701050 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:34.701656 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:34.701673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.702822 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.702827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:34.704185 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:34.704198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.705045 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.705052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:34.705339 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:34.705987 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:34.706009 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:34.708351 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:34.711588 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:34.711598 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:34.711615 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:34.712254 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:34.712730 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:34.712735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:34.712739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:34.712935 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:34.713444 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:34.713449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:34.713876 (+ 427us) client_negotiation.cc:769] Sending connection context
0619 15:02:34.714124 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 15:02:34.714519 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":385,"threads_started":1}
I20260619 15:02:34.717278 25498 heartbeater.cc:347] Connected to a master server at 127.24.103.126:33747
I20260619 15:02:34.717614 25498 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:34.718382 25498 heartbeater.cc:511] Master 127.24.103.126:33747 requested a full tablet report, sending...
I20260619 15:02:34.719946 25041 ts_manager.cc:194] Registered new tserver with Master: 888a0f18c6f64a5e8bea26b3469d2741 (127.24.103.67:33125)
I20260619 15:02:34.720819 25041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:55487
I20260619 15:02:34.725239 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 15:02:34 dist-test-slave-zpbw krb5kdc[24996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881354, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 15:02:34 dist-test-slave-zpbw krb5kdc[24996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881354, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:02:34.772771 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:34.756445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.24.103.65:38483)
0619 15:02:34.757074 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:34.757081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:34.757158 (+ 77us) server_negotiation.cc:407] Connection header received
0619 15:02:34.757328 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:34.757335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:34.757556 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:34.757698 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:34.759228 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.760251 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.761526 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.762124 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.762224 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:34.766495 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:34.766510 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:34.766527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:34.766577 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:34.769847 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:34.770614 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:34.770621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:34.770623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:34.770753 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:34.771220 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:34.771224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:34.771229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:34.771711 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:34.772010 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:34.772231 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 15:02:34.772452 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":323,"threads_started":1}
W20260619 15:02:34.774019 25145 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44908
Jun 19 15:02:34 dist-test-slave-zpbw krb5kdc[24996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 15:02:34 dist-test-slave-zpbw krb5kdc[24996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:02:34.820495 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:34.806009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.24.103.65:38483)
0619 15:02:34.806260 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:34.806267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:34.806487 (+ 220us) server_negotiation.cc:407] Connection header received
0619 15:02:34.806648 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:34.806653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:34.806724 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:34.806881 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:34.808375 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.809145 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.810459 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.810950 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.811052 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:34.815453 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:34.815458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:34.815462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:34.815501 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:34.817958 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:34.818602 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:34.818606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:34.818609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:34.818733 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:34.819188 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:34.819192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:34.819195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:34.819477 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:34.819670 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:34.819941 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 15:02:34.820128 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 15:02:34.822250 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 25102
I20260619 15:02:34.835693 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:34.823503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.24.103.65:38483)
0619 15:02:34.823723 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:34.823730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:34.823867 (+ 137us) server_negotiation.cc:407] Connection header received
0619 15:02:34.823997 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:34.824000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:34.824057 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:34.824136 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:34.825290 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.826419 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.827770 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.828193 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.828296 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:34.830520 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:34.830524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:34.830527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:34.830558 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:34.833300 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:34.833934 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:34.833938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:34.833940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:34.834067 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:34.834378 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:34.834381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:34.834382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:34.834630 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:34.834873 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:34.835135 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 15:02:34.835317 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 15:02:34.836476 25225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:02:34.925115 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 25102
I20260619 15:02:34.941138 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 25236
Jun 19 15:02:34 dist-test-slave-zpbw krb5kdc[24996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:02:34.957249 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:34.942056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.24.103.66:33951)
0619 15:02:34.942582 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:34.942589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:34.942615 (+ 26us) server_negotiation.cc:407] Connection header received
0619 15:02:34.942677 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:34.942683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:34.943002 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:34.943140 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:34.944389 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.945391 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.946643 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:34.947343 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:34.947443 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:34.951771 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:34.951785 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:34.951800 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:34.951836 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:34.954577 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:34.955347 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:34.955355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:34.955357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:34.955479 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:34.955840 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:34.955843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:34.955845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:34.956174 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:34.956406 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:34.956761 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 15:02:34.956964 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":266,"threads_started":1}
I20260619 15:02:34.958500 25359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:02:35.041786 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 25236
I20260619 15:02:35.057371 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 25370
Jun 19 15:02:35 dist-test-slave-zpbw krb5kdc[24996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:02:35.074240 25515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:35.058341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.24.103.67:33125)
0619 15:02:35.059053 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:35.059060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:35.059090 (+ 30us) server_negotiation.cc:407] Connection header received
0619 15:02:35.059147 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:35.059153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:35.059366 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:35.059488 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:35.060766 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:35.061852 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:35.063198 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:35.063795 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:35.063902 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:35.068335 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:35.068349 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:35.068365 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:35.068407 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:35.071490 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:35.072198 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:35.072205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:35.072207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:35.072334 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:35.072690 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:35.072695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:35.072697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:35.073085 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:35.073349 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:35.073591 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 15:02:35.073805 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":344,"threads_started":1}
I20260619 15:02:35.075794 25493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:02:35.158946 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 25370
I20260619 15:02:35.174157 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 25009
I20260619 15:02:35.175395 25071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:02:35.279084 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 25009
2026-06-19T15:02:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4027 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:02:35 dist-test-slave-zpbw krb5kdc[25520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:02:35 dist-test-slave-zpbw krb5kdc[25520](info): set up 2 sockets
Jun 19 15:02:35 dist-test-slave-zpbw krb5kdc[25520](info): commencing operation
krb5kdc: starting...
W20260619 15:02:36.363142 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:02:36 dist-test-slave-zpbw krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:02:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:02:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:02:36.544688 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42865
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:36.877568 25533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:36.877916 25533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:36.878083 25533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:36.889882 25533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:02:36.890007 25533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:36.890089 25533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:36.890172 25533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:02:36.890256 25533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:02:36.900661 25533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:36.902046 25533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:36.903484 25533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:36.916133 25539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:36.916180 25538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:36.916548 25541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:36.917462 25533 server_base.cc:1034] running on GCE node
I20260619 15:02:36.918298 25533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:36.919832 25533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:36.921074 25533 hybrid_clock.cc:648] HybridClock initialized: now 1781881356921010 us; error 35 us; skew 500 ppm
I20260619 15:02:36.921486 25533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:36 dist-test-slave-zpbw krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881356, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:36.924818 25533 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:02:36.926384 25533 webserver.cc:460] Webserver started at http://127.24.103.126:44847/ using document root <none> and password file <none>
I20260619 15:02:36.927183 25533 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:36.927340 25533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:36.927662 25533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:36.930738 25533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "eee8a88f475e4b2089e8c658f718a795"
format_stamp: "Formatted at 2026-06-19 15:02:36 on dist-test-slave-zpbw"
server_key: "b66a18b055f67e5e0b00afab17557112"
server_key_iv: "2fc34225b2a8f2b42133bb5c86f226cf"
server_key_version: "encryptionkey@0"
I20260619 15:02:36.931617 25533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "eee8a88f475e4b2089e8c658f718a795"
format_stamp: "Formatted at 2026-06-19 15:02:36 on dist-test-slave-zpbw"
server_key: "b66a18b055f67e5e0b00afab17557112"
server_key_iv: "2fc34225b2a8f2b42133bb5c86f226cf"
server_key_version: "encryptionkey@0"
I20260619 15:02:36.936906 25533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 15:02:36.940415 25548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:36.941821 25533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 15:02:36.941998 25533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "eee8a88f475e4b2089e8c658f718a795"
format_stamp: "Formatted at 2026-06-19 15:02:36 on dist-test-slave-zpbw"
server_key: "b66a18b055f67e5e0b00afab17557112"
server_key_iv: "2fc34225b2a8f2b42133bb5c86f226cf"
server_key_version: "encryptionkey@0"
I20260619 15:02:36.942204 25533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:36.967767 25533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:36.972090 25533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:36.972391 25533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:36.991328 25533 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:42865
I20260619 15:02:36.991356 25599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:42865 every 8 connection(s)
I20260619 15:02:36.992830 25533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:02:36.996770 25600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:37.003093 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 25533
I20260619 15:02:37.003321 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:02:37.003994 24989 external_mini_cluster.cc:1428] Setting key 9c40329a7fdc5474212a85813d7f5b38
I20260619 15:02:37.007900 25600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795: Bootstrap starting.
I20260619 15:02:37.011725 25600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:37.012774 25600 log.cc:824] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:37.015749 25600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795: No bootstrap required, opened a new log
Jun 19 15:02:37 dist-test-slave-zpbw krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:37.025944 25600 raft_consensus.cc:348] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee8a88f475e4b2089e8c658f718a795" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42865 } }
I20260619 15:02:37.026472 25600 raft_consensus.cc:374] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:37.026611 25600 raft_consensus.cc:729] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee8a88f475e4b2089e8c658f718a795, State: Initialized, Role: FOLLOWER
I20260619 15:02:37.027365 25600 consensus_queue.cc:260] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee8a88f475e4b2089e8c658f718a795" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42865 } }
I20260619 15:02:37.027616 25600 raft_consensus.cc:388] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:02:37.027809 25600 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:02:37.028002 25600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:37.028483 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:37.006091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.24.103.126:42865)
0619 15:02:37.006922 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:37.006934 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:37.006976 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:02:37.008741 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:37.008765 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:37.009158 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:37.009504 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:37.011436 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:37.012782 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:37.014405 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:37.015123 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:37.015343 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:37.019914 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:37.019922 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:37.019941 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:37.019984 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:37.023467 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:37.024470 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:37.024477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:37.024483 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:37.024683 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:37.025103 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:37.025106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:37.025108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:37.025688 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:37.025908 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:37.026607 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 15:02:37.027153 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":387,"threads_started":1}
I20260619 15:02:37.030979 25600 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee8a88f475e4b2089e8c658f718a795" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42865 } }
I20260619 15:02:37.031587 25600 leader_election.cc:302] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eee8a88f475e4b2089e8c658f718a795; no voters:
I20260619 15:02:37.032194 25600 leader_election.cc:288] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:02:37.032327 25605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:37.033094 25605 raft_consensus.cc:686] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [term 1 LEADER]: Becoming Leader. State: Replica: eee8a88f475e4b2089e8c658f718a795, State: Running, Role: LEADER
I20260619 15:02:37.033666 25605 consensus_queue.cc:237] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee8a88f475e4b2089e8c658f718a795" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42865 } }
I20260619 15:02:37.034403 25600 sys_catalog.cc:564] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:02:37.036759 25607 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee8a88f475e4b2089e8c658f718a795. Latest consensus state: current_term: 1 leader_uuid: "eee8a88f475e4b2089e8c658f718a795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee8a88f475e4b2089e8c658f718a795" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42865 } } }
I20260619 15:02:37.037171 25607 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [sys.catalog]: This master's current role is: LEADER
I20260619 15:02:37.038570 25606 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eee8a88f475e4b2089e8c658f718a795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee8a88f475e4b2089e8c658f718a795" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42865 } } }
I20260619 15:02:37.039095 25606 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795 [sys.catalog]: This master's current role is: LEADER
I20260619 15:02:37.048619 25610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:02:37.055927 25610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:02:37.067222 25610 catalog_manager.cc:1347] Generated new cluster ID: d2358bac2a5d41e7bf046e01cf631904
I20260619 15:02:37.067350 25610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:02:37.081995 25610 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:02:37.083514 25610 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:02:37.092406 25610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eee8a88f475e4b2089e8c658f718a795: Generated new TSK 0
I20260619 15:02:37.093314 25610 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:02:37.134212 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42865
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:37.465953 25627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:37.466276 25627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:37.466375 25627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:37.473810 25627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:37.473933 25627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:37.474051 25627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:02:37.484334 25627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42865
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:37.485646 25627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:37.487008 25627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:37.500946 25632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:37.500902 25635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:37.500877 25633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:37.502758 25627 server_base.cc:1034] running on GCE node
I20260619 15:02:37.503525 25627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:37.504612 25627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:37.505813 25627 hybrid_clock.cc:648] HybridClock initialized: now 1781881357505769 us; error 37 us; skew 500 ppm
I20260619 15:02:37.506155 25627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:37 dist-test-slave-zpbw krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881357, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:37.509507 25627 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:02:37.511189 25627 webserver.cc:460] Webserver started at http://127.24.103.65:41097/ using document root <none> and password file <none>
I20260619 15:02:37.511916 25627 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:37.512059 25627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:37.512382 25627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:37.515501 25627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "065424ce8a344513a04d51f6af541f60"
format_stamp: "Formatted at 2026-06-19 15:02:37 on dist-test-slave-zpbw"
server_key: "752fbad46320a17442a18cd43fc0cc96"
server_key_iv: "ee1615130640bfb55407a624a433bcb6"
server_key_version: "encryptionkey@0"
I20260619 15:02:37.516309 25627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "065424ce8a344513a04d51f6af541f60"
format_stamp: "Formatted at 2026-06-19 15:02:37 on dist-test-slave-zpbw"
server_key: "752fbad46320a17442a18cd43fc0cc96"
server_key_iv: "ee1615130640bfb55407a624a433bcb6"
server_key_version: "encryptionkey@0"
I20260619 15:02:37.521803 25627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:02:37.525574 25642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:37.527046 25627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:02:37.527226 25627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "065424ce8a344513a04d51f6af541f60"
format_stamp: "Formatted at 2026-06-19 15:02:37 on dist-test-slave-zpbw"
server_key: "752fbad46320a17442a18cd43fc0cc96"
server_key_iv: "ee1615130640bfb55407a624a433bcb6"
server_key_version: "encryptionkey@0"
I20260619 15:02:37.527434 25627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:37.539786 25627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:37.544250 25627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:37.544559 25627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:37.545569 25627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:02:37.547118 25627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:37.547217 25627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:37.547370 25627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:37.547441 25627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:37.580991 25627 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:34595
I20260619 15:02:37.581039 25754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:34595 every 8 connection(s)
I20260619 15:02:37.582381 25627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:02:37.592768 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 25627
I20260619 15:02:37.592985 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:02:37.593755 24989 external_mini_cluster.cc:1428] Setting key 5f0590fe490a8b5e688ba6fe15eae6bc
Jun 19 15:02:37 dist-test-slave-zpbw krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881357, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:37.604745 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:37.586708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:36005 (local address 127.24.103.126:42865)
0619 15:02:37.587179 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:37.587185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:37.587208 (+ 23us) server_negotiation.cc:407] Connection header received
0619 15:02:37.589061 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:37.589066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:37.589132 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:37.589234 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:37.590866 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:37.591587 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:37.593376 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:37.594068 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:37.594207 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:37.598222 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:37.598229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:37.598237 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:37.598285 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:37.601297 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:37.602317 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:37.602321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:37.602323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:37.602436 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:37.602952 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:37.602956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:37.602957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:37.603258 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:37.603379 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:37.604299 (+ 920us) server_negotiation.cc:299] Negotiation successful
0619 15:02:37.604478 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":242,"threads_started":1}
I20260619 15:02:37.606365 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:37.586143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42865 (local address 127.24.103.65:36005)
0619 15:02:37.587037 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:37.587112 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:37.588806 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:37.589504 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:37.589517 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:37.590036 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:37.590655 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:37.590672 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:37.591736 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:37.591740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:37.593228 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:37.593244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:37.594276 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:37.594281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:37.594523 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:37.595491 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:37.595514 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:37.597965 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:37.601450 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:37.601462 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:37.601501 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:37.602199 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:37.602544 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:37.602551 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:37.602558 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:37.602839 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:37.603402 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:37.603413 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:37.603903 (+ 490us) client_negotiation.cc:769] Sending connection context
0619 15:02:37.604223 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 15:02:37.604832 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":294,"threads_started":1}
I20260619 15:02:37.608103 25755 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42865
I20260619 15:02:37.608419 25755 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:37.609134 25755 heartbeater.cc:511] Master 127.24.103.126:42865 requested a full tablet report, sending...
I20260619 15:02:37.611337 25565 ts_manager.cc:194] Registered new tserver with Master: 065424ce8a344513a04d51f6af541f60 (127.24.103.65:34595)
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260619 15:02:37.612732 25565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:36005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:02:37.634063 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42865
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:37.958470 25761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:37.958835 25761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:37.959038 25761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:37.966454 25761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:37.966590 25761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:37.966708 25761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:02:37.976511 25761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42865
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:37.977728 25761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:37.978994 25761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:37.992336 25767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:37.992555 25769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:37.992857 25766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:37.994908 25761 server_base.cc:1034] running on GCE node
I20260619 15:02:37.995657 25761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:37.996835 25761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:37.998124 25761 hybrid_clock.cc:648] HybridClock initialized: now 1781881357998050 us; error 87 us; skew 500 ppm
I20260619 15:02:37.998474 25761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:38 dist-test-slave-zpbw krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881358, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:38.006259 25761 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:02:38.007997 25761 webserver.cc:460] Webserver started at http://127.24.103.66:34849/ using document root <none> and password file <none>
I20260619 15:02:38.008718 25761 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:38.008854 25761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:38.009192 25761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:38.012128 25761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "92f72ceb26f94ece90fa2fad63aad1eb"
format_stamp: "Formatted at 2026-06-19 15:02:38 on dist-test-slave-zpbw"
server_key: "932aba9c4d5c99d4a4012473afa5a969"
server_key_iv: "4b783423eea5440cf779ec8c70968a80"
server_key_version: "encryptionkey@0"
I20260619 15:02:38.012904 25761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "92f72ceb26f94ece90fa2fad63aad1eb"
format_stamp: "Formatted at 2026-06-19 15:02:38 on dist-test-slave-zpbw"
server_key: "932aba9c4d5c99d4a4012473afa5a969"
server_key_iv: "4b783423eea5440cf779ec8c70968a80"
server_key_version: "encryptionkey@0"
I20260619 15:02:38.018146 25761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 15:02:38.022367 25776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:38.024139 25761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:02:38.024343 25761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "92f72ceb26f94ece90fa2fad63aad1eb"
format_stamp: "Formatted at 2026-06-19 15:02:38 on dist-test-slave-zpbw"
server_key: "932aba9c4d5c99d4a4012473afa5a969"
server_key_iv: "4b783423eea5440cf779ec8c70968a80"
server_key_version: "encryptionkey@0"
I20260619 15:02:38.024575 25761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:38.056209 25761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:38.060874 25761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:38.061177 25761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:38.062244 25761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:02:38.063655 25761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:38.063769 25761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:38.063923 25761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:38.063994 25761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:38.097465 25761 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:35209
I20260619 15:02:38.097508 25888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:35209 every 8 connection(s)
I20260619 15:02:38.098841 25761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:02:38.102748 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 25761
I20260619 15:02:38.102986 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:02:38.103464 24989 external_mini_cluster.cc:1428] Setting key b90090b66776b3fe8e2b0e59858f8343
Jun 19 15:02:38 dist-test-slave-zpbw krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881358, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260619 15:02:38.121229 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.101446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:35451 (local address 127.24.103.126:42865)
0619 15:02:38.101622 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:38.101627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:38.102897 (+ 1270us) server_negotiation.cc:407] Connection header received
0619 15:02:38.105059 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:38.105064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:38.105145 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:38.105248 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:38.107390 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.108144 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.110581 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.111056 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.111209 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.115141 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:38.115147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:38.115150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:38.115184 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:38.117953 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:38.118735 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:38.118742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:38.118750 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:38.118954 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:38.119394 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:38.119399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:38.119402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:38.119685 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:38.119841 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:38.120668 (+ 827us) server_negotiation.cc:299] Negotiation successful
0619 15:02:38.120886 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 15:02:38.122350 25891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.101973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42865 (local address 127.24.103.66:35451)
0619 15:02:38.102676 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:38.102718 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:38.104798 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:38.105534 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:38.105546 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:38.106311 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:38.107176 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.107206 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.108313 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.108322 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.110420 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.110432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.111327 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.111336 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.111575 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.112308 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:38.112348 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:38.114911 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:38.118129 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:38.118140 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:38.118158 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:38.118599 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:38.119068 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:38.119073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:38.119077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:38.119272 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:38.119813 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:38.119821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:38.120252 (+ 431us) client_negotiation.cc:769] Sending connection context
0619 15:02:38.120537 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 15:02:38.121008 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":299,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
I20260619 15:02:38.123950 25889 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42865
I20260619 15:02:38.124303 25889 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:38.125207 25889 heartbeater.cc:511] Master 127.24.103.126:42865 requested a full tablet report, sending...
I20260619 15:02:38.126921 25565 ts_manager.cc:194] Registered new tserver with Master: 92f72ceb26f94ece90fa2fad63aad1eb (127.24.103.66:35209)
I20260619 15:02:38.128139 25565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:35451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:02:38.143419 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42865
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:38.471552 25894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:38.471916 25894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:38.472095 25894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:38.479488 25894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:38.479673 25894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:38.479858 25894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:02:38.490026 25894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42865
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:38.491400 25894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:38.492887 25894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:38.505897 25899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:38.506021 25900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:38.506089 25902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:38.506932 25894 server_base.cc:1034] running on GCE node
I20260619 15:02:38.507563 25894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:38.508559 25894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:38.509793 25894 hybrid_clock.cc:648] HybridClock initialized: now 1781881358509725 us; error 63 us; skew 500 ppm
I20260619 15:02:38.510108 25894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:38 dist-test-slave-zpbw krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881358, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:38.513953 25894 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:02:38.515654 25894 webserver.cc:460] Webserver started at http://127.24.103.67:40989/ using document root <none> and password file <none>
I20260619 15:02:38.516350 25894 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:38.516470 25894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:38.516754 25894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:38.519987 25894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "4dfc287a9dbe4b61965b08a4c24b1099"
format_stamp: "Formatted at 2026-06-19 15:02:38 on dist-test-slave-zpbw"
server_key: "d56e95013a3c66ae6dde88f21183f67d"
server_key_iv: "8ac3b16460b791796f1d50834eb36c5d"
server_key_version: "encryptionkey@0"
I20260619 15:02:38.520823 25894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "4dfc287a9dbe4b61965b08a4c24b1099"
format_stamp: "Formatted at 2026-06-19 15:02:38 on dist-test-slave-zpbw"
server_key: "d56e95013a3c66ae6dde88f21183f67d"
server_key_iv: "8ac3b16460b791796f1d50834eb36c5d"
server_key_version: "encryptionkey@0"
I20260619 15:02:38.526539 25894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:02:38.530598 25909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:38.532320 25894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:02:38.532493 25894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "4dfc287a9dbe4b61965b08a4c24b1099"
format_stamp: "Formatted at 2026-06-19 15:02:38 on dist-test-slave-zpbw"
server_key: "d56e95013a3c66ae6dde88f21183f67d"
server_key_iv: "8ac3b16460b791796f1d50834eb36c5d"
server_key_version: "encryptionkey@0"
I20260619 15:02:38.532734 25894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:38.558826 25894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:38.563223 25894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:38.563504 25894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:38.564458 25894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:02:38.565768 25894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:38.565845 25894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:38.565927 25894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:38.565971 25894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:38.602473 25894 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:38769
I20260619 15:02:38.602512 26021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:38769 every 8 connection(s)
I20260619 15:02:38.603868 25894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:02:38.611636 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 25894
I20260619 15:02:38.611869 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:02:38.612277 24989 external_mini_cluster.cc:1428] Setting key ff44bf2b10164c8447f4a2d83ba9dc57
I20260619 15:02:38.615859 25755 heartbeater.cc:503] Master 127.24.103.126:42865 was elected leader, sending a full tablet report...
Jun 19 15:02:38 dist-test-slave-zpbw krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881358, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:38.625594 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.606592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:60769 (local address 127.24.103.126:42865)
0619 15:02:38.606828 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:38.606835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:38.608057 (+ 1222us) server_negotiation.cc:407] Connection header received
0619 15:02:38.610095 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:38.610101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:38.610217 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:38.610353 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:38.612121 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.613116 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.615313 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.616017 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.616171 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.619817 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:38.619821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:38.619824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:38.619856 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:38.622532 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:38.623580 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:38.623584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:38.623587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:38.623714 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:38.624122 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:38.624125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:38.624127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:38.624390 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:38.624496 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:38.625183 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 15:02:38.625352 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:02:38.627050 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.607089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42865 (local address 127.24.103.67:60769)
0619 15:02:38.607871 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:38.607912 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:38.609794 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:38.610606 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:38.610615 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:38.611231 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:38.611904 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.611923 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.613320 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.613325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.615107 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.615121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.616256 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.616265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.616520 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.617287 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:38.617316 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:38.619605 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:38.622688 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:38.622696 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:38.622710 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:38.623337 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:38.623829 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:38.623833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:38.623835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:38.624019 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:38.624518 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:38.624524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:38.624866 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 15:02:38.625101 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 15:02:38.625506 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":299,"threads_started":1}
I20260619 15:02:38.628659 26022 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42865
I20260619 15:02:38.628966 26022 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:38.629685 26022 heartbeater.cc:511] Master 127.24.103.126:42865 requested a full tablet report, sending...
I20260619 15:02:38.631179 25564 ts_manager.cc:194] Registered new tserver with Master: 4dfc287a9dbe4b61965b08a4c24b1099 (127.24.103.67:38769)
I20260619 15:02:38.632158 25564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:60769
I20260619 15:02:38.641131 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:02:38.661343 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.647755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.24.103.126:42865)
0619 15:02:38.647924 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:38.647928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:38.648608 (+ 680us) server_negotiation.cc:407] Connection header received
0619 15:02:38.648685 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:38.648689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:38.648761 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:38.648861 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:38.650719 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.651659 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.652908 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.653403 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.653520 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.655874 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:38.655879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:38.655882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:38.655917 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:38.658625 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:38.659404 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:38.659409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:38.659411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:38.659566 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:38.659936 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:38.659945 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:38.659953 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:38.660330 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:38.660541 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:38.660837 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 15:02:38.661095 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 15:02:38.668959 25564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:02:38.677275 25564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:02:38.724092 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.691014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:35209 (local address 127.0.0.1:38672)
0619 15:02:38.714549 (+ 23535us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:38.714573 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:38.714725 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:38.716589 (+ 1864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:38.716597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:38.716626 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:38.717025 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.717038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.718957 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.718963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.721155 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.721167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.723037 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.723042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.723228 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.723290 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 15:02:38.723462 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 15:02:38.723659 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23417,"mutex_wait_us":26,"thread_start_us":23798,"threads_started":1}
I20260619 15:02:38.724062 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.691553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.24.103.66:35209)
0619 15:02:38.713030 (+ 21477us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:38.713043 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:38.714712 (+ 1669us) server_negotiation.cc:407] Connection header received
0619 15:02:38.715034 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:38.715048 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:38.715459 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:38.715714 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:38.717190 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.718770 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.721313 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.722435 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.722566 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.722813 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:38.723628 (+ 815us) server_negotiation.cc:299] Negotiation successful
0619 15:02:38.723798 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21238,"thread_start_us":21523,"threads_started":1}
I20260619 15:02:38.725574 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.690458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:38769 (local address 127.0.0.1:48274)
0619 15:02:38.707675 (+ 17217us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:38.707720 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:38.707936 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:38.717788 (+ 9852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:38.717794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:38.717821 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:38.718280 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.718292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.720150 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.720155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.722483 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.722502 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.724754 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.724761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.724928 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.724962 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 15:02:38.725056 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 15:02:38.725175 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16945,"mutex_wait_us":1,"thread_start_us":16746,"threads_started":1}
I20260619 15:02:38.725661 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.690712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.24.103.67:38769)
0619 15:02:38.717260 (+ 26548us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:38.717266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:38.717290 (+ 24us) server_negotiation.cc:407] Connection header received
0619 15:02:38.717359 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:38.717363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:38.717573 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:38.717778 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:38.718456 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.720001 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.723077 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.724461 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.724619 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.724838 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:38.725116 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 15:02:38.725292 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26366,"thread_start_us":26190,"threads_started":1}
I20260619 15:02:38.726176 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.691303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.24.103.65:34595)
0619 15:02:38.716905 (+ 25602us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:38.716917 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:38.716950 (+ 33us) server_negotiation.cc:407] Connection header received
0619 15:02:38.717042 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:38.717049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:38.717318 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:38.717509 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:38.718186 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.719719 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.722494 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.724503 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.724623 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.724841 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:38.725663 (+ 822us) server_negotiation.cc:299] Negotiation successful
0619 15:02:38.725837 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25381,"thread_start_us":25209,"threads_started":1}
I20260619 15:02:38.726419 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.690639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34595 (local address 127.0.0.1:38212)
0619 15:02:38.713817 (+ 23178us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:38.713845 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:38.714031 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:38.717517 (+ 3486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:38.717521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:38.717544 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:38.717962 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.717975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.719851 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.719858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.722290 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.722305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.724971 (+ 2666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.724976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.725173 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.725207 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 15:02:38.725300 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 15:02:38.725436 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22989,"mutex_wait_us":23,"thread_start_us":23674,"threads_started":1}
I20260619 15:02:38.728901 25824 tablet_service.cc:1467] Processing CreateTablet for tablet a9637a68d7c84c91bf1ac5168614c1a0 (DEFAULT_TABLE table=test-table [id=b5cc2b28a6434bdcb1b4c00cee32eb15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:38.729550 25957 tablet_service.cc:1467] Processing CreateTablet for tablet a9637a68d7c84c91bf1ac5168614c1a0 (DEFAULT_TABLE table=test-table [id=b5cc2b28a6434bdcb1b4c00cee32eb15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:38.730471 25824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9637a68d7c84c91bf1ac5168614c1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:38.730949 25957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9637a68d7c84c91bf1ac5168614c1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:38.738344 25690 tablet_service.cc:1467] Processing CreateTablet for tablet a9637a68d7c84c91bf1ac5168614c1a0 (DEFAULT_TABLE table=test-table [id=b5cc2b28a6434bdcb1b4c00cee32eb15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:38.740098 25690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9637a68d7c84c91bf1ac5168614c1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:38.745966 26040 tablet_bootstrap.cc:492] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099: Bootstrap starting.
I20260619 15:02:38.749261 26041 tablet_bootstrap.cc:492] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb: Bootstrap starting.
I20260619 15:02:38.750705 26040 tablet_bootstrap.cc:654] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:38.751818 26040 log.cc:824] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:38.752873 26041 tablet_bootstrap.cc:654] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:38.754132 26041 log.cc:824] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:38.755319 26040 tablet_bootstrap.cc:492] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099: No bootstrap required, opened a new log
I20260619 15:02:38.755690 26040 ts_tablet_manager.cc:1397] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260619 15:02:38.758428 26044 tablet_bootstrap.cc:492] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60: Bootstrap starting.
I20260619 15:02:38.759588 26041 tablet_bootstrap.cc:492] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb: No bootstrap required, opened a new log
I20260619 15:02:38.759874 26041 ts_tablet_manager.cc:1397] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260619 15:02:38.762188 26044 tablet_bootstrap.cc:654] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:38.763939 26044 log.cc:824] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:38.766723 26040 raft_consensus.cc:348] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38769 } } peers { permanent_uuid: "065424ce8a344513a04d51f6af541f60" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34595 } } peers { permanent_uuid: "92f72ceb26f94ece90fa2fad63aad1eb" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35209 } }
I20260619 15:02:38.767539 26040 raft_consensus.cc:374] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:38.767611 26044 tablet_bootstrap.cc:492] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60: No bootstrap required, opened a new log
I20260619 15:02:38.767746 26040 raft_consensus.cc:729] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dfc287a9dbe4b61965b08a4c24b1099, State: Initialized, Role: FOLLOWER
I20260619 15:02:38.767911 26044 ts_tablet_manager.cc:1397] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 15:02:38.767970 26041 raft_consensus.cc:348] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38769 } } peers { permanent_uuid: "065424ce8a344513a04d51f6af541f60" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34595 } } peers { permanent_uuid: "92f72ceb26f94ece90fa2fad63aad1eb" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35209 } }
I20260619 15:02:38.768482 26041 raft_consensus.cc:374] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:38.768606 26041 raft_consensus.cc:729] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f72ceb26f94ece90fa2fad63aad1eb, State: Initialized, Role: FOLLOWER
I20260619 15:02:38.768738 26040 consensus_queue.cc:260] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38769 } } peers { permanent_uuid: "065424ce8a344513a04d51f6af541f60" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34595 } } peers { permanent_uuid: "92f72ceb26f94ece90fa2fad63aad1eb" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35209 } }
I20260619 15:02:38.769212 26041 consensus_queue.cc:260] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38769 } } peers { permanent_uuid: "065424ce8a344513a04d51f6af541f60" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34595 } } peers { permanent_uuid: "92f72ceb26f94ece90fa2fad63aad1eb" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35209 } }
I20260619 15:02:38.769927 26022 heartbeater.cc:503] Master 127.24.103.126:42865 was elected leader, sending a full tablet report...
I20260619 15:02:38.770725 26040 ts_tablet_manager.cc:1428] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260619 15:02:38.771816 25889 heartbeater.cc:503] Master 127.24.103.126:42865 was elected leader, sending a full tablet report...
I20260619 15:02:38.776039 26041 ts_tablet_manager.cc:1428] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260619 15:02:38.779477 26044 raft_consensus.cc:348] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38769 } } peers { permanent_uuid: "065424ce8a344513a04d51f6af541f60" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34595 } } peers { permanent_uuid: "92f72ceb26f94ece90fa2fad63aad1eb" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35209 } }
I20260619 15:02:38.779950 26044 raft_consensus.cc:374] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:38.780081 26044 raft_consensus.cc:729] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065424ce8a344513a04d51f6af541f60, State: Initialized, Role: FOLLOWER
I20260619 15:02:38.780856 26044 consensus_queue.cc:260] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38769 } } peers { permanent_uuid: "065424ce8a344513a04d51f6af541f60" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34595 } } peers { permanent_uuid: "92f72ceb26f94ece90fa2fad63aad1eb" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35209 } }
I20260619 15:02:38.782441 26044 ts_tablet_manager.cc:1428] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260619 15:02:38.835791 25756 tablet.cc:2369] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:02:38.843604 26046 raft_consensus.cc:482] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:02:38.843930 26046 raft_consensus.cc:504] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38769 } } peers { permanent_uuid: "065424ce8a344513a04d51f6af541f60" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34595 } } peers { permanent_uuid: "92f72ceb26f94ece90fa2fad63aad1eb" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35209 } }
I20260619 15:02:38.845808 26046 leader_election.cc:288] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 065424ce8a344513a04d51f6af541f60 (127.24.103.65:34595), 92f72ceb26f94ece90fa2fad63aad1eb (127.24.103.66:35209)
W20260619 15:02:38.851147 25890 tablet.cc:2369] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:02:38.852933 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.846132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:51099 (local address 127.24.103.65:34595)
0619 15:02:38.846363 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:38.846371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:38.846398 (+ 27us) server_negotiation.cc:407] Connection header received
0619 15:02:38.846629 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:38.846635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:38.846725 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:38.846895 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:38.847534 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.848848 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.850856 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.851935 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.852128 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.852202 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:38.852564 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 15:02:38.852684 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 15:02:38.852952 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.846046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34595 (local address 127.24.103.67:51099)
0619 15:02:38.846265 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:38.846291 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:38.846424 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:38.847089 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:38.847097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:38.847155 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:38.847359 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.847371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.848990 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.848996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.850681 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.850693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.852115 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.852122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.852409 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.852432 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 15:02:38.852537 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 15:02:38.852671 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"spinlock_wait_cycles":896}
I20260619 15:02:38.853803 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.846610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:45987 (local address 127.24.103.66:35209)
0619 15:02:38.846849 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:38.846857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:38.847119 (+ 262us) server_negotiation.cc:407] Connection header received
0619 15:02:38.847277 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:38.847282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:38.847361 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:38.847485 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:38.848519 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.849655 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.851651 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.852761 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.852965 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.853039 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:38.853397 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 15:02:38.853529 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:02:38.853963 26049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.846233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:35209 (local address 127.24.103.67:45987)
0619 15:02:38.847010 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:38.847024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:38.847147 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:38.847520 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:38.847524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:38.847546 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:38.848347 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.848360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.849813 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.849817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.851445 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:38.851458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.852979 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.852985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:38.853187 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.853233 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 15:02:38.853368 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 15:02:38.853512 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"spinlock_wait_cycles":21120,"thread_start_us":426,"threads_started":1}
I20260619 15:02:38.854223 25710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9637a68d7c84c91bf1ac5168614c1a0" candidate_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "065424ce8a344513a04d51f6af541f60" is_pre_election: true
I20260619 15:02:38.854864 25710 raft_consensus.cc:2454] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dfc287a9dbe4b61965b08a4c24b1099 in term 0.
I20260619 15:02:38.855041 25844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9637a68d7c84c91bf1ac5168614c1a0" candidate_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f72ceb26f94ece90fa2fad63aad1eb" is_pre_election: true
I20260619 15:02:38.855611 25844 raft_consensus.cc:2454] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dfc287a9dbe4b61965b08a4c24b1099 in term 0.
W20260619 15:02:38.855697 26023 tablet.cc:2369] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:02:38.855671 25912 leader_election.cc:302] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065424ce8a344513a04d51f6af541f60, 4dfc287a9dbe4b61965b08a4c24b1099; no voters:
I20260619 15:02:38.856144 26046 raft_consensus.cc:2781] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:02:38.856323 26046 raft_consensus.cc:482] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:02:38.856465 26046 raft_consensus.cc:3037] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:38.858870 26046 raft_consensus.cc:504] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38769 } } peers { permanent_uuid: "065424ce8a344513a04d51f6af541f60" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34595 } } peers { permanent_uuid: "92f72ceb26f94ece90fa2fad63aad1eb" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35209 } }
I20260619 15:02:38.859786 26046 leader_election.cc:288] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [CANDIDATE]: Term 1 election: Requested vote from peers 065424ce8a344513a04d51f6af541f60 (127.24.103.65:34595), 92f72ceb26f94ece90fa2fad63aad1eb (127.24.103.66:35209)
I20260619 15:02:38.860340 25844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9637a68d7c84c91bf1ac5168614c1a0" candidate_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f72ceb26f94ece90fa2fad63aad1eb"
I20260619 15:02:38.860335 25710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9637a68d7c84c91bf1ac5168614c1a0" candidate_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "065424ce8a344513a04d51f6af541f60"
I20260619 15:02:38.860594 25710 raft_consensus.cc:3037] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:38.860622 25844 raft_consensus.cc:3037] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:38.862944 25710 raft_consensus.cc:2454] T a9637a68d7c84c91bf1ac5168614c1a0 P 065424ce8a344513a04d51f6af541f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dfc287a9dbe4b61965b08a4c24b1099 in term 1.
I20260619 15:02:38.863416 25912 leader_election.cc:302] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065424ce8a344513a04d51f6af541f60, 4dfc287a9dbe4b61965b08a4c24b1099; no voters:
I20260619 15:02:38.863713 26046 raft_consensus.cc:2781] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:38.863713 25844 raft_consensus.cc:2454] T a9637a68d7c84c91bf1ac5168614c1a0 P 92f72ceb26f94ece90fa2fad63aad1eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dfc287a9dbe4b61965b08a4c24b1099 in term 1.
I20260619 15:02:38.864286 26046 raft_consensus.cc:686] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [term 1 LEADER]: Becoming Leader. State: Replica: 4dfc287a9dbe4b61965b08a4c24b1099, State: Running, Role: LEADER
I20260619 15:02:38.864827 26046 consensus_queue.cc:237] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38769 } } peers { permanent_uuid: "065424ce8a344513a04d51f6af541f60" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34595 } } peers { permanent_uuid: "92f72ceb26f94ece90fa2fad63aad1eb" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35209 } }
I20260619 15:02:38.869912 25565 catalog_manager.cc:5495] T a9637a68d7c84c91bf1ac5168614c1a0 P 4dfc287a9dbe4b61965b08a4c24b1099 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dfc287a9dbe4b61965b08a4c24b1099 (127.24.103.67). New cstate: current_term: 1 leader_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dfc287a9dbe4b61965b08a4c24b1099" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "065424ce8a344513a04d51f6af541f60" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92f72ceb26f94ece90fa2fad63aad1eb" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35209 } health_report { overall_health: UNKNOWN } } }
Jun 19 15:02:38 dist-test-slave-zpbw krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:02:38.926038 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.910262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.24.103.65:34595)
0619 15:02:38.910499 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:38.910509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:38.910958 (+ 449us) server_negotiation.cc:407] Connection header received
0619 15:02:38.911105 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:38.911110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:38.911173 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:38.911259 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:38.912921 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.913653 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.915108 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.915543 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.915637 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.920022 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:38.920039 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:38.920055 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:38.920125 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:38.923313 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:38.924111 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:38.924118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:38.924120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:38.924254 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:38.924652 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:38.924655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:38.924657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:38.925006 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:38.925210 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:38.925512 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 15:02:38.925671 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 15:02:38 dist-test-slave-zpbw krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881358, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 15:02:38 dist-test-slave-zpbw krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881358, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:02:38.974731 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:38.961434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.24.103.65:34595)
0619 15:02:38.961678 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:38.961686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:38.961864 (+ 178us) server_negotiation.cc:407] Connection header received
0619 15:02:38.962017 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:38.962022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:38.962086 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:38.962173 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:38.963628 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.964368 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.965683 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:38.966089 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:38.966191 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:38.970128 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:38.970134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:38.970137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:38.970171 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:38.972561 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:38.973175 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:38.973179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:38.973181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:38.973294 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:38.973586 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:38.973589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:38.973591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:38.973872 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:38.974075 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:38.974243 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 15:02:38.974405 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 15:02:38 dist-test-slave-zpbw krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 15:02:39 dist-test-slave-zpbw krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:02:39.020305 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:39.005943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.24.103.65:34595)
0619 15:02:39.006189 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:39.006196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:39.006401 (+ 205us) server_negotiation.cc:407] Connection header received
0619 15:02:39.006618 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:39.006623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:39.006685 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:39.006837 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:39.008202 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:39.008890 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:39.010142 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:39.010574 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:39.010717 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:39.015027 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:39.015033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:39.015036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:39.015074 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:39.017527 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:39.018269 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:39.018277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:39.018285 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:39.018411 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:39.018808 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:39.018814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:39.018820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:39.019130 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:39.019395 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:39.019695 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 15:02:39.019937 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260619 15:02:39.022087 25670 tablet_service.cc:2908] Rejecting scan request for tablet a9637a68d7c84c91bf1ac5168614c1a0: Uninitialized: clean time has not yet been initialized
I20260619 15:02:39.025262 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 25627
I20260619 15:02:39.038944 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:39.026199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.24.103.65:34595)
0619 15:02:39.026420 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:39.026430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:39.026708 (+ 278us) server_negotiation.cc:407] Connection header received
0619 15:02:39.026994 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:39.026998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:39.027072 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:39.027177 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:39.028405 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:39.029076 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:39.030302 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:39.030770 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:39.030962 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:39.033255 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:39.033263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:39.033274 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:39.033306 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:39.036032 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:39.036688 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:39.036693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:39.036695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:39.036821 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:39.037186 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:39.037190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:39.037192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:39.037542 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:39.037749 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:39.037987 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 15:02:39.038226 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 15:02:39.039595 25750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:02:39.137622 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 25627
I20260619 15:02:39.154224 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 25761
Jun 19 15:02:39 dist-test-slave-zpbw krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:02:39.169247 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:39.155148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.24.103.66:35209)
0619 15:02:39.155329 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:39.155333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:39.155373 (+ 40us) server_negotiation.cc:407] Connection header received
0619 15:02:39.155430 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:39.155433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:39.155497 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:39.155626 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:39.156790 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:39.157547 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:39.158857 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:39.159338 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:39.159441 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:39.163782 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:39.163798 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:39.163814 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:39.163864 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:39.166605 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:39.167468 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:39.167475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:39.167477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:39.167620 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:39.167963 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:39.167967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:39.167969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:39.168354 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:39.168578 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:39.168812 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 15:02:39.168983 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 15:02:39.169952 25884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:02:39.257481 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 25761
I20260619 15:02:39.273917 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 25894
Jun 19 15:02:39 dist-test-slave-zpbw krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:02:39.289222 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:39.274686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.24.103.67:38769)
0619 15:02:39.275196 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:39.275200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:39.275216 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:02:39.275278 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:39.275283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:39.275339 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:39.275461 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:39.276788 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:39.277465 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:39.278756 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:39.279319 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:39.279464 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:39.283672 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:39.283689 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:39.283706 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:39.283743 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:39.286289 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:39.287109 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:39.287116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:39.287119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:39.287259 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:39.287659 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:39.287669 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:39.287671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:39.287987 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:39.288300 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:39.288583 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 15:02:39.288811 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":246,"threads_started":1}
I20260619 15:02:39.289872 26017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:02:39.381228 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 25894
I20260619 15:02:39.397553 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 25533
I20260619 15:02:39.398715 25595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:02:39.528779 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 25533
2026-06-19T15:02:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4247 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:02:39 dist-test-slave-zpbw krb5kdc[26072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:02:39 dist-test-slave-zpbw krb5kdc[26072](info): set up 2 sockets
Jun 19 15:02:39 dist-test-slave-zpbw krb5kdc[26072](info): commencing operation
krb5kdc: starting...
W20260619 15:02:40.607234 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:02:40 dist-test-slave-zpbw krb5kdc[26072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:02:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:02:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:02:40.790658 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:45433
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:45433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:41.132680 26085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:41.133010 26085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:41.133219 26085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:41.140735 26085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:02:41.140863 26085 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 15:02:41.140996 26085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:41.141131 26085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:41.141227 26085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:02:41.141357 26085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:02:41.151965 26085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:45433
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:45433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:41.153384 26085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:41.154836 26085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:41.168078 26090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:41.167985 26093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:41.168012 26091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:41.169188 26085 server_base.cc:1034] running on GCE node
I20260619 15:02:41.170012 26085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:41.171602 26085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:41.172870 26085 hybrid_clock.cc:648] HybridClock initialized: now 1781881361172790 us; error 57 us; skew 500 ppm
I20260619 15:02:41.173225 26085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:41 dist-test-slave-zpbw krb5kdc[26072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881361, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:41.176885 26085 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:02:41.178591 26085 webserver.cc:460] Webserver started at http://127.24.103.126:46195/ using document root <none> and password file <none>
I20260619 15:02:41.179466 26085 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:41.179605 26085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:41.179960 26085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:41.183095 26085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "66ad46b02f4e402c821e646e09d09035"
format_stamp: "Formatted at 2026-06-19 15:02:41 on dist-test-slave-zpbw"
server_key: "8c38d424c3844dbd8626c513e02e8ec4"
server_key_iv: "b37e5969207229f9574f2d84de514bf6"
server_key_version: "encryptionkey@0"
I20260619 15:02:41.183889 26085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "66ad46b02f4e402c821e646e09d09035"
format_stamp: "Formatted at 2026-06-19 15:02:41 on dist-test-slave-zpbw"
server_key: "8c38d424c3844dbd8626c513e02e8ec4"
server_key_iv: "b37e5969207229f9574f2d84de514bf6"
server_key_version: "encryptionkey@0"
I20260619 15:02:41.189316 26085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 15:02:41.192955 26100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:41.194377 26085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:02:41.194603 26085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "66ad46b02f4e402c821e646e09d09035"
format_stamp: "Formatted at 2026-06-19 15:02:41 on dist-test-slave-zpbw"
server_key: "8c38d424c3844dbd8626c513e02e8ec4"
server_key_iv: "b37e5969207229f9574f2d84de514bf6"
server_key_version: "encryptionkey@0"
I20260619 15:02:41.194901 26085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:41.215723 26085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:41.220316 26085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:41.220551 26085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:41.242981 26085 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:45433
I20260619 15:02:41.243018 26161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:45433 every 8 connection(s)
I20260619 15:02:41.244513 26085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:02:41.248479 26162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:41.249444 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 26085
I20260619 15:02:41.249706 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:02:41.250517 24989 external_mini_cluster.cc:1428] Setting key a612fe0ee9ae6797ac0cef39ca04a4ee
I20260619 15:02:41.263862 26162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035: Bootstrap starting.
Jun 19 15:02:41 dist-test-slave-zpbw krb5kdc[26072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:41.268007 26162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:41.269471 26162 log.cc:824] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:41.273157 26162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035: No bootstrap required, opened a new log
I20260619 15:02:41.276384 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:41.252775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54356 (local address 127.24.103.126:45433)
0619 15:02:41.253627 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:41.253642 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:41.253687 (+ 45us) server_negotiation.cc:407] Connection header received
0619 15:02:41.255502 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:41.255531 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:41.255978 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:41.256341 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:41.258233 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:41.259670 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:41.261574 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:41.262396 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:41.262570 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:41.267672 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:41.267684 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:41.267704 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:41.267762 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:41.271759 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:41.272487 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:41.272498 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:41.272509 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:41.272732 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:41.273087 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:41.273092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:41.273095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:41.273835 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:41.274147 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:41.274481 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 15:02:41.274961 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":390,"threads_started":1}
I20260619 15:02:41.282486 26162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ad46b02f4e402c821e646e09d09035" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45433 } }
I20260619 15:02:41.282974 26162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:41.283105 26162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ad46b02f4e402c821e646e09d09035, State: Initialized, Role: FOLLOWER
I20260619 15:02:41.283741 26162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ad46b02f4e402c821e646e09d09035" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45433 } }
I20260619 15:02:41.283972 26162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:02:41.284113 26162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:02:41.284279 26162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:41.286504 26162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ad46b02f4e402c821e646e09d09035" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45433 } }
I20260619 15:02:41.287081 26162 leader_election.cc:302] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66ad46b02f4e402c821e646e09d09035; no voters:
I20260619 15:02:41.287739 26162 leader_election.cc:288] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:02:41.287904 26167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:41.289106 26167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [term 1 LEADER]: Becoming Leader. State: Replica: 66ad46b02f4e402c821e646e09d09035, State: Running, Role: LEADER
I20260619 15:02:41.289665 26167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ad46b02f4e402c821e646e09d09035" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45433 } }
I20260619 15:02:41.290053 26162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:02:41.292656 26168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66ad46b02f4e402c821e646e09d09035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ad46b02f4e402c821e646e09d09035" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45433 } } }
I20260619 15:02:41.293131 26168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [sys.catalog]: This master's current role is: LEADER
I20260619 15:02:41.293671 26169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66ad46b02f4e402c821e646e09d09035. Latest consensus state: current_term: 1 leader_uuid: "66ad46b02f4e402c821e646e09d09035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ad46b02f4e402c821e646e09d09035" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45433 } } }
I20260619 15:02:41.293946 26169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035 [sys.catalog]: This master's current role is: LEADER
I20260619 15:02:41.295606 26175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:02:41.301139 26175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:02:41.310746 26175 catalog_manager.cc:1347] Generated new cluster ID: b9b32d61cdae43f28a35a1f400e97fb1
I20260619 15:02:41.310941 26175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:02:41.319844 26175 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:02:41.321341 26175 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:02:41.328162 26175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035: Generated new TSK 0
I20260619 15:02:41.329232 26175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:02:41.372378 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45433
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:41.716343 26188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:41.716643 26188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:41.716804 26188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:41.724241 26188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:41.724362 26188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:41.724525 26188 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 15:02:41.724663 26188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:02:41.734848 26188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:45433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:41.736246 26188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:41.737625 26188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:41.751145 26193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:41.751246 26196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:41.751226 26194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:41.752010 26188 server_base.cc:1034] running on GCE node
I20260619 15:02:41.752600 26188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:41.753615 26188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:41.754913 26188 hybrid_clock.cc:648] HybridClock initialized: now 1781881361754824 us; error 64 us; skew 500 ppm
I20260619 15:02:41.755300 26188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:41 dist-test-slave-zpbw krb5kdc[26072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881361, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:41.758706 26188 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:02:41.760346 26188 webserver.cc:460] Webserver started at http://127.24.103.65:45669/ using document root <none> and password file <none>
I20260619 15:02:41.761109 26188 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:41.761230 26188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:41.761585 26188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:41.764745 26188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "d9390f6a2af746e78d4a654085261f6b"
format_stamp: "Formatted at 2026-06-19 15:02:41 on dist-test-slave-zpbw"
server_key: "482c0e7920180edaa23565d3b95052aa"
server_key_iv: "369fcf98bea82e1844a43b9cc19161dc"
server_key_version: "encryptionkey@0"
I20260619 15:02:41.765630 26188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "d9390f6a2af746e78d4a654085261f6b"
format_stamp: "Formatted at 2026-06-19 15:02:41 on dist-test-slave-zpbw"
server_key: "482c0e7920180edaa23565d3b95052aa"
server_key_iv: "369fcf98bea82e1844a43b9cc19161dc"
server_key_version: "encryptionkey@0"
I20260619 15:02:41.771232 26188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 15:02:41.775031 26203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:41.776844 26188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:02:41.777048 26188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "d9390f6a2af746e78d4a654085261f6b"
format_stamp: "Formatted at 2026-06-19 15:02:41 on dist-test-slave-zpbw"
server_key: "482c0e7920180edaa23565d3b95052aa"
server_key_iv: "369fcf98bea82e1844a43b9cc19161dc"
server_key_version: "encryptionkey@0"
I20260619 15:02:41.777390 26188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:41.792557 26188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:41.798018 26188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:41.798358 26188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:41.801460 26188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:41.801589 26188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:41.801800 26188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:41.801926 26188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 15:02:41 dist-test-slave-zpbw krb5kdc[26072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881361, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:41.824824 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:41.801842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:48675 (local address 127.24.103.126:45433)
0619 15:02:41.802330 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:41.802337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:41.803630 (+ 1293us) server_negotiation.cc:407] Connection header received
0619 15:02:41.805694 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:41.805700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:41.805799 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:41.805917 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:41.808181 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:41.809242 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:41.811816 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:41.812455 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:41.812609 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:41.817957 (+ 5348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:41.817962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:41.817966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:41.818000 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:41.821214 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:41.822320 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:41.822326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:41.822330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:41.822469 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:41.823001 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:41.823006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:41.823014 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:41.823314 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:41.823474 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:41.824212 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 15:02:41.824439 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":240,"threads_started":1}
I20260619 15:02:41.826020 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:41.802477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45433 (local address 127.24.103.65:48675)
0619 15:02:41.803396 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:41.803466 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:41.805389 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:41.806173 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:41.806181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:41.806983 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:41.807972 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:41.808005 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:41.809425 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:41.809429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:41.811639 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:41.811651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:41.812739 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:41.812747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:41.813035 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:41.813958 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:41.814000 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:41.817696 (+ 3696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:41.821427 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:41.821436 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:41.821454 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:41.822182 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:41.822603 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:41.822608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:41.822612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:41.822873 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:41.823458 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:41.823466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:41.823853 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 15:02:41.824120 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 15:02:41.824679 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":448,"threads_started":1}
I20260619 15:02:41.850627 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:41.833476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:35291 (local address 127.24.103.126:45433)
0619 15:02:41.833680 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:41.833684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:41.834292 (+ 608us) server_negotiation.cc:407] Connection header received
0619 15:02:41.834554 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:41.834565 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:41.834634 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:41.834746 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:41.836271 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:41.837421 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:41.839674 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:41.840318 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:41.840494 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:41.843391 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:41.843397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:41.843401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:41.843435 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:41.846991 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:41.848112 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:41.848120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:41.848127 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:41.848305 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:41.848816 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:41.848822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:41.848826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:41.849260 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:41.849461 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:41.849978 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 15:02:41.850222 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 15:02:41.850760 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:41.833486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45433 (local address 127.24.103.65:35291)
0619 15:02:41.834175 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:41.834196 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:41.834370 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:41.834808 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:41.834814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:41.835687 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:41.836106 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:41.836117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:41.837580 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:41.837587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:41.839449 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:41.839474 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:41.840575 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:41.840584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:41.840788 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:41.841727 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:41.841763 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:41.843184 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:41.847191 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:41.847205 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:41.847208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:41.847913 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:41.848463 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:41.848467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:41.848469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:41.848647 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:41.849424 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:41.849451 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:41.849698 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 15:02:41.849868 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 15:02:41.850233 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":350,"threads_started":1}
I20260619 15:02:41.856992 26188 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:37253
I20260619 15:02:41.857076 26323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:37253 every 8 connection(s)
I20260619 15:02:41.858213 26188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:02:41.860390 26324 heartbeater.cc:347] Connected to a master server at 127.24.103.126:45433
I20260619 15:02:41.860625 26324 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:41.861390 26324 heartbeater.cc:511] Master 127.24.103.126:45433 requested a full tablet report, sending...
I20260619 15:02:41.861703 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 26188
I20260619 15:02:41.861922 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:02:41.862730 24989 external_mini_cluster.cc:1428] Setting key 620624530a3224f0881f4ff9937a7880
I20260619 15:02:41.863770 26117 ts_manager.cc:194] Registered new tserver with Master: d9390f6a2af746e78d4a654085261f6b (127.24.103.65:37253)
I20260619 15:02:41.865376 26117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:48675
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:02:41.903909 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45433
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:42.225903 26328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:42.226197 26328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:42.226289 26328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:42.233675 26328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:42.233791 26328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:42.233876 26328 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 15:02:42.233958 26328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:02:42.243976 26328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:45433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:42.245263 26328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:42.246654 26328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:42.260447 26336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:42.260432 26334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:42.261015 26333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:42.261260 26328 server_base.cc:1034] running on GCE node
I20260619 15:02:42.261996 26328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:42.263056 26328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:42.264344 26328 hybrid_clock.cc:648] HybridClock initialized: now 1781881362264286 us; error 57 us; skew 500 ppm
I20260619 15:02:42.264675 26328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:42 dist-test-slave-zpbw krb5kdc[26072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881362, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:42.268033 26328 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:02:42.269498 26328 webserver.cc:460] Webserver started at http://127.24.103.66:45037/ using document root <none> and password file <none>
I20260619 15:02:42.270211 26328 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:42.270344 26328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:42.270639 26328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:42.273607 26328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "ea3d79bc059d42afad732c04c48e9a54"
format_stamp: "Formatted at 2026-06-19 15:02:42 on dist-test-slave-zpbw"
server_key: "761395d9ce2e7d21df163db0f277576a"
server_key_iv: "13cb555eef9eff35ff648972e4651cbc"
server_key_version: "encryptionkey@0"
I20260619 15:02:42.274531 26328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "ea3d79bc059d42afad732c04c48e9a54"
format_stamp: "Formatted at 2026-06-19 15:02:42 on dist-test-slave-zpbw"
server_key: "761395d9ce2e7d21df163db0f277576a"
server_key_iv: "13cb555eef9eff35ff648972e4651cbc"
server_key_version: "encryptionkey@0"
I20260619 15:02:42.279958 26328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 15:02:42.283716 26343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:42.285187 26328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:02:42.285375 26328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "ea3d79bc059d42afad732c04c48e9a54"
format_stamp: "Formatted at 2026-06-19 15:02:42 on dist-test-slave-zpbw"
server_key: "761395d9ce2e7d21df163db0f277576a"
server_key_iv: "13cb555eef9eff35ff648972e4651cbc"
server_key_version: "encryptionkey@0"
I20260619 15:02:42.285632 26328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:42.302106 26328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:42.306555 26328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:42.306908 26328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:42.309525 26328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:42.309649 26328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:42.309816 26328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:42.309899 26328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 15:02:42 dist-test-slave-zpbw krb5kdc[26072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881362, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:42.336675 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.310349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:43019 (local address 127.24.103.126:45433)
0619 15:02:42.310523 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:42.310529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:42.312123 (+ 1594us) server_negotiation.cc:407] Connection header received
0619 15:02:42.315367 (+ 3244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:42.315375 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:42.315464 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:42.315596 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:42.318026 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.319146 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.321532 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.322186 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.322350 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.328370 (+ 6020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:42.328377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:42.328381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:42.328425 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:42.331929 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:42.333307 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:42.333314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:42.333318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:42.333500 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:42.334014 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:42.334021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:42.334024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:42.334441 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:42.334598 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:42.335784 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0619 15:02:42.336145 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 15:02:42.337383 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.310999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45433 (local address 127.24.103.66:43019)
0619 15:02:42.311884 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:42.311958 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:42.314876 (+ 2918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:42.315898 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:42.315915 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:42.316755 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:42.317805 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.317839 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.319425 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.319431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.321373 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.321386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.322478 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.322488 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.322894 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.323905 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:42.323951 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:42.327935 (+ 3984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:42.332144 (+ 4209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:42.332161 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:42.332192 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:42.333134 (+ 942us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:42.333631 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:42.333640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:42.333644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:42.333886 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:42.334662 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:42.334668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:42.335278 (+ 610us) client_negotiation.cc:769] Sending connection context
0619 15:02:42.335648 (+ 370us) client_negotiation.cc:240] Negotiation successful
0619 15:02:42.336131 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":318,"threads_started":1}
I20260619 15:02:42.357784 26462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:41111 every 8 connection(s)
I20260619 15:02:42.357780 26328 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:41111
I20260619 15:02:42.359587 26328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:02:42.361661 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.343690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45433 (local address 127.24.103.66:34169)
0619 15:02:42.344824 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:42.344849 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:42.345060 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:42.345522 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:42.345530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:42.346429 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:42.347059 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.347080 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.348679 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.348686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.349987 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.350000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.351476 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.351486 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.351815 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.352698 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:42.352733 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:42.354354 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:42.358667 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:42.358671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:42.358675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:42.359147 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:42.359573 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:42.359577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:42.359580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:42.359763 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:42.360581 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:42.360586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:42.360820 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 15:02:42.360969 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 15:02:42.361276 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":711,"threads_started":1}
I20260619 15:02:42.362064 26463 heartbeater.cc:347] Connected to a master server at 127.24.103.126:45433
I20260619 15:02:42.362259 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 26328
I20260619 15:02:42.362313 26463 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:42.362475 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:02:42.362674 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.343564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:34169 (local address 127.24.103.126:45433)
0619 15:02:42.343771 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:42.343793 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:42.344950 (+ 1157us) server_negotiation.cc:407] Connection header received
0619 15:02:42.345216 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:42.345223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:42.345322 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:42.345434 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:42.347314 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.348499 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.350156 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.350992 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.351136 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.354664 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:42.354671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:42.354678 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:42.354731 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:42.358508 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:42.359274 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:42.359279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:42.359282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:42.359444 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:42.359852 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:42.359858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:42.359861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:42.360382 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:42.361970 (+ 1588us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:42.362093 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 15:02:42.362330 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 15:02:42.363070 24989 external_mini_cluster.cc:1428] Setting key 5c39bff3e404570bf53c179ad85d7d40
I20260619 15:02:42.363171 26463 heartbeater.cc:511] Master 127.24.103.126:45433 requested a full tablet report, sending...
I20260619 15:02:42.364754 26115 ts_manager.cc:194] Registered new tserver with Master: ea3d79bc059d42afad732c04c48e9a54 (127.24.103.66:41111)
I20260619 15:02:42.365629 26115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:43019
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:02:42.402148 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45433
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:42.732350 26467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:42.732695 26467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:42.732864 26467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:42.740909 26467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:42.741078 26467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:42.741221 26467 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 15:02:42.741366 26467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:02:42.751652 26467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:45433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:42.753047 26467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:42.754393 26467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:42.767584 26472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:42.767539 26475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:42.767650 26473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:42.768428 26467 server_base.cc:1034] running on GCE node
I20260619 15:02:42.769014 26467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:42.769894 26467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:42.771152 26467 hybrid_clock.cc:648] HybridClock initialized: now 1781881362771070 us; error 58 us; skew 500 ppm
I20260619 15:02:42.771462 26467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:42 dist-test-slave-zpbw krb5kdc[26072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881362, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:42.774658 26467 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:02:42.776212 26467 webserver.cc:460] Webserver started at http://127.24.103.67:37573/ using document root <none> and password file <none>
I20260619 15:02:42.776907 26467 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:42.776995 26467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:42.777276 26467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:42.780258 26467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "56c2eabc75fd43349318a25de0dfca1f"
format_stamp: "Formatted at 2026-06-19 15:02:42 on dist-test-slave-zpbw"
server_key: "4dbdc0fd8be12eb0622a7708333073aa"
server_key_iv: "1129724299d78144332554aa6b350fc2"
server_key_version: "encryptionkey@0"
I20260619 15:02:42.781118 26467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "56c2eabc75fd43349318a25de0dfca1f"
format_stamp: "Formatted at 2026-06-19 15:02:42 on dist-test-slave-zpbw"
server_key: "4dbdc0fd8be12eb0622a7708333073aa"
server_key_iv: "1129724299d78144332554aa6b350fc2"
server_key_version: "encryptionkey@0"
I20260619 15:02:42.786607 26467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 15:02:42.790311 26482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:42.791960 26467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:02:42.792146 26467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "56c2eabc75fd43349318a25de0dfca1f"
format_stamp: "Formatted at 2026-06-19 15:02:42 on dist-test-slave-zpbw"
server_key: "4dbdc0fd8be12eb0622a7708333073aa"
server_key_iv: "1129724299d78144332554aa6b350fc2"
server_key_version: "encryptionkey@0"
I20260619 15:02:42.792387 26467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:42.805298 26467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:42.809847 26467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:42.810127 26467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:42.812711 26467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:42.812832 26467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:42.812966 26467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:42.813031 26467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 15:02:42 dist-test-slave-zpbw krb5kdc[26072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881362, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:42.840884 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.813580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:39673 (local address 127.24.103.126:45433)
0619 15:02:42.813753 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:42.813758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:42.815331 (+ 1573us) server_negotiation.cc:407] Connection header received
0619 15:02:42.817745 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:42.817750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:42.817860 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:42.817947 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:42.820596 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.821661 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.825140 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.825601 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.825760 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.831557 (+ 5797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:42.831564 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:42.831572 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:42.831630 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:42.835907 (+ 4277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:42.837030 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:42.837036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:42.837038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:42.837164 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:42.837886 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:42.837896 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:42.837902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:42.838401 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:42.838560 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:42.840075 (+ 1515us) server_negotiation.cc:299] Negotiation successful
0619 15:02:42.840381 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:02:42.842715 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.814230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45433 (local address 127.24.103.67:39673)
0619 15:02:42.815156 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:42.815195 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:42.817326 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:42.818504 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:42.818523 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:42.819452 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:42.820390 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.820412 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.821871 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.821882 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.824945 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.824969 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.825860 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.825867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.826265 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.827269 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:42.827313 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:42.831216 (+ 3903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:42.836109 (+ 4893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:42.836125 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:42.836148 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:42.836841 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:42.837317 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:42.837326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:42.837332 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:42.837705 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:42.838635 (+ 930us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:42.838645 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:42.839361 (+ 716us) client_negotiation.cc:769] Sending connection context
0619 15:02:42.839804 (+ 443us) client_negotiation.cc:240] Negotiation successful
0619 15:02:42.840620 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":388,"threads_started":1}
I20260619 15:02:42.856528 26467 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:36837
I20260619 15:02:42.856489 26601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:36837 every 8 connection(s)
I20260619 15:02:42.858018 26467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:02:42.860571 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 26467
I20260619 15:02:42.860796 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:02:42.861246 26602 heartbeater.cc:347] Connected to a master server at 127.24.103.126:45433
I20260619 15:02:42.861392 24989 external_mini_cluster.cc:1428] Setting key 6797ead7a1cb049a48005d22191a5980
I20260619 15:02:42.861639 26602 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:42.862716 26602 heartbeater.cc:511] Master 127.24.103.126:45433 requested a full tablet report, sending...
I20260619 15:02:42.867070 26115 ts_manager.cc:194] Registered new tserver with Master: 56c2eabc75fd43349318a25de0dfca1f (127.24.103.67:36837)
I20260619 15:02:42.868265 26115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:39673
I20260619 15:02:42.869119 26324 heartbeater.cc:503] Master 127.24.103.126:45433 was elected leader, sending a full tablet report...
I20260619 15:02:42.870818 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.850134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:37957 (local address 127.24.103.126:45433)
0619 15:02:42.850327 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:42.850333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:42.852279 (+ 1946us) server_negotiation.cc:407] Connection header received
0619 15:02:42.852556 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:42.852563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:42.852680 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:42.852808 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:42.854492 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.855577 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.857593 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.858266 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.858414 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.861473 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:42.861483 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:42.861492 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:42.861547 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:42.866517 (+ 4970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:42.867791 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:42.867800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:42.867806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:42.868007 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:42.868502 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:42.868510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:42.868516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:42.869561 (+ 1045us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:42.869751 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:42.870209 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 15:02:42.870440 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 15:02:42.870973 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.849705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45433 (local address 127.24.103.67:37957)
0619 15:02:42.852127 (+ 2422us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:42.852152 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:42.852384 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:42.852874 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:42.852879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:42.853779 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:42.854308 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.854326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.855721 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.855734 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.857414 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.857431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.858486 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.858493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.858733 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.859611 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:42.859644 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:42.861191 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:42.866736 (+ 5545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:42.866746 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:42.866753 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:42.867563 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:42.868170 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:42.868175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:42.868180 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:42.868353 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:42.869711 (+ 1358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:42.869716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:42.869938 (+ 222us) client_negotiation.cc:769] Sending connection context
0619 15:02:42.870099 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 15:02:42.870435 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2149,"thread_start_us":1860,"threads_started":1}
I20260619 15:02:42.878575 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 15:02:42 dist-test-slave-zpbw krb5kdc[26072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 15:02:42 dist-test-slave-zpbw krb5kdc[26072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:42.929782 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.914909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.24.103.126:45433)
0619 15:02:42.915085 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:42.915089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:42.915427 (+ 338us) server_negotiation.cc:407] Connection header received
0619 15:02:42.915600 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:42.915604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:42.915669 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:42.915764 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:42.917198 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.917969 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.919407 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.919897 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.919999 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.924445 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:42.924450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:42.924454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:42.924491 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:42.927177 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:42.927908 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:42.927913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:42.927915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:42.928049 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:42.928400 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:42.928404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:42.928406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:42.928748 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:42.928934 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:42.929237 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 15:02:42.929419 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 15:02:42.934180 26115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:02:42.941509 26115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:02:42.970101 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.954401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.24.103.66:41111)
0619 15:02:42.955395 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:42.955408 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:42.956247 (+ 839us) server_negotiation.cc:407] Connection header received
0619 15:02:42.956367 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:42.956378 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:42.956688 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:42.956904 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:42.959142 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.962083 (+ 2941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.966902 (+ 4819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.968576 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.969288 (+ 712us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.969513 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:42.969635 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 15:02:42.969805 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":735,"threads_started":1}
I20260619 15:02:42.969843 26615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.954265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:41111 (local address 127.0.0.1:41690)
0619 15:02:42.955814 (+ 1549us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:42.955852 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:42.956044 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:42.958179 (+ 2135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:42.958190 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:42.958230 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:42.958793 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.958806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.962279 (+ 3473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.962288 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.966622 (+ 4334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.966659 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.968827 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.968836 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.969102 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.969164 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 15:02:42.969289 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 15:02:42.969479 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1293,"thread_start_us":449,"threads_started":1}
I20260619 15:02:42.974701 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.954956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:36837 (local address 127.0.0.1:52986)
0619 15:02:42.966960 (+ 12004us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:42.966988 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:42.967140 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:42.968110 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:42.968115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:42.968148 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:42.968507 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.968519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.970449 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.970454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.972350 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.972361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.973936 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.973942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.974093 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.974127 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 15:02:42.974220 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 15:02:42.974341 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11845,"thread_start_us":12412,"threads_started":1}
I20260619 15:02:42.974810 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.955160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.24.103.67:36837)
0619 15:02:42.966379 (+ 11219us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:42.966393 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:42.967428 (+ 1035us) server_negotiation.cc:407] Connection header received
0619 15:02:42.967552 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:42.967564 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:42.967880 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:42.968100 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:42.968753 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.970302 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.972509 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.973694 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.973859 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.974072 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:42.974279 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 15:02:42.974434 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10971,"thread_start_us":10753,"threads_started":1}
I20260619 15:02:42.974581 26393 tablet_service.cc:1467] Processing CreateTablet for tablet 19545c4172424eb49c6b5846963e7dae (DEFAULT_TABLE table=test-table [id=3cad265e871941da85090cee4349ae55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:42.975768 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.955658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37253 (local address 127.0.0.1:52858)
0619 15:02:42.967191 (+ 11533us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:42.967209 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:42.967301 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:42.968591 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:42.968595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:42.968622 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:42.969049 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.969057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.971176 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.971180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.973119 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:42.973133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.975107 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.975113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:42.975271 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.975296 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 15:02:42.975358 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 15:02:42.975447 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11424,"thread_start_us":11568,"threads_started":1}
I20260619 15:02:42.976430 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:42.955811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.24.103.65:37253)
0619 15:02:42.967989 (+ 12178us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:42.967999 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:42.968041 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:02:42.968134 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:42.968143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:42.968397 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:42.968602 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:42.969220 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.971019 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.973293 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:42.974711 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:42.974889 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:42.975872 (+ 983us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:42.975939 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 15:02:42.976104 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12008,"thread_start_us":11873,"threads_started":1}
I20260619 15:02:42.977384 26393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19545c4172424eb49c6b5846963e7dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:42.978848 26532 tablet_service.cc:1467] Processing CreateTablet for tablet 19545c4172424eb49c6b5846963e7dae (DEFAULT_TABLE table=test-table [id=3cad265e871941da85090cee4349ae55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:42.979759 26254 tablet_service.cc:1467] Processing CreateTablet for tablet 19545c4172424eb49c6b5846963e7dae (DEFAULT_TABLE table=test-table [id=3cad265e871941da85090cee4349ae55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:42.980284 26532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19545c4172424eb49c6b5846963e7dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:42.980996 26254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19545c4172424eb49c6b5846963e7dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:42.988780 26621 tablet_bootstrap.cc:492] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54: Bootstrap starting.
I20260619 15:02:42.992522 26621 tablet_bootstrap.cc:654] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:42.993814 26621 log.cc:824] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:42.994446 26622 tablet_bootstrap.cc:492] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f: Bootstrap starting.
I20260619 15:02:42.998335 26621 tablet_bootstrap.cc:492] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54: No bootstrap required, opened a new log
I20260619 15:02:42.998409 26622 tablet_bootstrap.cc:654] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:42.998622 26621 ts_tablet_manager.cc:1397] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 15:02:42.998752 26624 tablet_bootstrap.cc:492] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b: Bootstrap starting.
I20260619 15:02:42.999799 26622 log.cc:824] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:43.002209 26624 tablet_bootstrap.cc:654] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:43.002996 26622 tablet_bootstrap.cc:492] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f: No bootstrap required, opened a new log
I20260619 15:02:43.003317 26622 ts_tablet_manager.cc:1397] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 15:02:43.003480 26624 log.cc:824] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:43.009678 26624 tablet_bootstrap.cc:492] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b: No bootstrap required, opened a new log
I20260619 15:02:43.009992 26624 ts_tablet_manager.cc:1397] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20260619 15:02:43.010767 26621 raft_consensus.cc:348] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } }
I20260619 15:02:43.011221 26621 raft_consensus.cc:374] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:43.011351 26621 raft_consensus.cc:729] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3d79bc059d42afad732c04c48e9a54, State: Initialized, Role: FOLLOWER
I20260619 15:02:43.012101 26621 consensus_queue.cc:260] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } }
I20260619 15:02:43.014699 26622 raft_consensus.cc:348] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } }
I20260619 15:02:43.015204 26622 raft_consensus.cc:374] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:43.015534 26463 heartbeater.cc:503] Master 127.24.103.126:45433 was elected leader, sending a full tablet report...
I20260619 15:02:43.016074 26621 ts_tablet_manager.cc:1428] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260619 15:02:43.016310 26622 raft_consensus.cc:729] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c2eabc75fd43349318a25de0dfca1f, State: Initialized, Role: FOLLOWER
I20260619 15:02:43.017108 26622 consensus_queue.cc:260] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } }
I20260619 15:02:43.020709 26622 ts_tablet_manager.cc:1428] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260619 15:02:43.020820 26624 raft_consensus.cc:348] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } }
I20260619 15:02:43.021049 26602 heartbeater.cc:503] Master 127.24.103.126:45433 was elected leader, sending a full tablet report...
I20260619 15:02:43.021313 26624 raft_consensus.cc:374] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:43.021422 26624 raft_consensus.cc:729] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9390f6a2af746e78d4a654085261f6b, State: Initialized, Role: FOLLOWER
I20260619 15:02:43.022070 26624 consensus_queue.cc:260] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } }
I20260619 15:02:43.023792 26624 ts_tablet_manager.cc:1428] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
W20260619 15:02:43.110754 26325 tablet.cc:2369] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:02:43.111847 26464 tablet.cc:2369] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:02:43.122637 26603 tablet.cc:2369] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:02:43.300123 26629 raft_consensus.cc:482] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:02:43.300565 26629 raft_consensus.cc:504] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } }
I20260619 15:02:43.302371 26629 leader_election.cc:288] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea3d79bc059d42afad732c04c48e9a54 (127.24.103.66:41111), 56c2eabc75fd43349318a25de0dfca1f (127.24.103.67:36837)
I20260619 15:02:43.309901 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:43.302993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:60499 (local address 127.24.103.66:41111)
0619 15:02:43.303188 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:43.303194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:43.303798 (+ 604us) server_negotiation.cc:407] Connection header received
0619 15:02:43.304089 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:43.304095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:43.304176 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:43.304262 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:43.305023 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.305780 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.308118 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.308838 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.309033 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:43.309151 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:43.309498 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 15:02:43.309629 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 15:02:43.309912 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:43.303190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:35799 (local address 127.24.103.67:36837)
0619 15:02:43.303378 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:43.303384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:43.304062 (+ 678us) server_negotiation.cc:407] Connection header received
0619 15:02:43.304282 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:43.304288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:43.304369 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:43.304479 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:43.305104 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.305776 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.308088 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.308826 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.308999 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:43.309147 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:43.309497 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 15:02:43.309626 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 15:02:43.310103 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:43.302860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:36837 (local address 127.24.103.65:35799)
0619 15:02:43.303932 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:43.303946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:43.304095 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:43.304501 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:43.304504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:43.304558 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:43.304914 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:43.304923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.305919 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.305926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:43.307916 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:43.307927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.309092 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.309098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:43.309299 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:43.309362 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 15:02:43.309468 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 15:02:43.309629 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":45,"spinlock_wait_cycles":23936,"thread_start_us":502,"threads_started":1}
I20260619 15:02:43.310459 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:43.302628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:41111 (local address 127.24.103.65:60499)
0619 15:02:43.303661 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:43.303686 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:43.303911 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:43.304313 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:43.304318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:43.304351 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:43.304852 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:43.304865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.305914 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.305921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:43.307929 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:43.307941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.309169 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.309178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:43.309309 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:43.309366 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 15:02:43.309469 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 15:02:43.309626 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":60,"thread_start_us":1051,"threads_started":1}
I20260619 15:02:43.311271 26552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19545c4172424eb49c6b5846963e7dae" candidate_uuid: "d9390f6a2af746e78d4a654085261f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c2eabc75fd43349318a25de0dfca1f" is_pre_election: true
I20260619 15:02:43.311457 26413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19545c4172424eb49c6b5846963e7dae" candidate_uuid: "d9390f6a2af746e78d4a654085261f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d79bc059d42afad732c04c48e9a54" is_pre_election: true
I20260619 15:02:43.311756 26552 raft_consensus.cc:2454] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9390f6a2af746e78d4a654085261f6b in term 0.
I20260619 15:02:43.312026 26413 raft_consensus.cc:2454] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9390f6a2af746e78d4a654085261f6b in term 0.
I20260619 15:02:43.312531 26206 leader_election.cc:302] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c2eabc75fd43349318a25de0dfca1f, d9390f6a2af746e78d4a654085261f6b; no voters:
I20260619 15:02:43.312922 26629 raft_consensus.cc:2781] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:02:43.313037 26629 raft_consensus.cc:482] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:02:43.313122 26629 raft_consensus.cc:3037] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:43.315373 26629 raft_consensus.cc:504] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } }
I20260619 15:02:43.315887 26629 leader_election.cc:288] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [CANDIDATE]: Term 1 election: Requested vote from peers ea3d79bc059d42afad732c04c48e9a54 (127.24.103.66:41111), 56c2eabc75fd43349318a25de0dfca1f (127.24.103.67:36837)
I20260619 15:02:43.316391 26552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19545c4172424eb49c6b5846963e7dae" candidate_uuid: "d9390f6a2af746e78d4a654085261f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c2eabc75fd43349318a25de0dfca1f"
I20260619 15:02:43.316391 26413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19545c4172424eb49c6b5846963e7dae" candidate_uuid: "d9390f6a2af746e78d4a654085261f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d79bc059d42afad732c04c48e9a54"
I20260619 15:02:43.316589 26413 raft_consensus.cc:3037] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:43.316638 26552 raft_consensus.cc:3037] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:43.320191 26413 raft_consensus.cc:2454] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9390f6a2af746e78d4a654085261f6b in term 1.
I20260619 15:02:43.320189 26552 raft_consensus.cc:2454] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9390f6a2af746e78d4a654085261f6b in term 1.
I20260619 15:02:43.320626 26205 leader_election.cc:302] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9390f6a2af746e78d4a654085261f6b, ea3d79bc059d42afad732c04c48e9a54; no voters:
I20260619 15:02:43.320981 26629 raft_consensus.cc:2781] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:43.321441 26629 raft_consensus.cc:686] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 1 LEADER]: Becoming Leader. State: Replica: d9390f6a2af746e78d4a654085261f6b, State: Running, Role: LEADER
I20260619 15:02:43.321950 26629 consensus_queue.cc:237] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } }
I20260619 15:02:43.326365 26115 catalog_manager.cc:5495] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b reported cstate change: term changed from 0 to 1, leader changed from <none> to d9390f6a2af746e78d4a654085261f6b (127.24.103.65). New cstate: current_term: 1 leader_uuid: "d9390f6a2af746e78d4a654085261f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } health_report { overall_health: HEALTHY } } }
I20260619 15:02:43.427385 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:43.421947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.24.103.65:37253)
0619 15:02:43.422130 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:43.422134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:43.422152 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:02:43.422399 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:43.422403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:43.422498 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:43.422619 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:02:43.423010 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.423717 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.425384 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.425761 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.425859 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:43.426292 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:02:43.426863 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:02:43.426971 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:43.427106 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 15:02:43.427172 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 15:02:43.435906 26413 raft_consensus.cc:1261] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54 [term 1 FOLLOWER]: Refusing update from remote peer d9390f6a2af746e78d4a654085261f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:43.435906 26552 raft_consensus.cc:1261] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f [term 1 FOLLOWER]: Refusing update from remote peer d9390f6a2af746e78d4a654085261f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:43.436915 26629 consensus_queue.cc:1035] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:43.437248 26632 consensus_queue.cc:1035] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:43.476245 26116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:02:43.476586 26116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:02:43.480525 26116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035: Sending DeleteTablet for 3 replicas of tablet 19545c4172424eb49c6b5846963e7dae
I20260619 15:02:43.481799 26532 tablet_service.cc:1514] Processing DeleteTablet for tablet 19545c4172424eb49c6b5846963e7dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:43 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:52986
I20260619 15:02:43.481805 26393 tablet_service.cc:1514] Processing DeleteTablet for tablet 19545c4172424eb49c6b5846963e7dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:43 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:41690
I20260619 15:02:43.483110 26645 tablet_replica.cc:330] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f: stopping tablet replica
I20260619 15:02:43.483047 26254 tablet_service.cc:1514] Processing DeleteTablet for tablet 19545c4172424eb49c6b5846963e7dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:43 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:52858
I20260619 15:02:43.483726 26645 raft_consensus.cc:2229] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:02:43.484337 26645 raft_consensus.cc:2258] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:43.485487 26116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:02:43.486434 26116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:02:43.493960 26647 tablet_replica.cc:330] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b: stopping tablet replica
I20260619 15:02:43.494648 26647 raft_consensus.cc:2229] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:02:43.495664 26647 raft_consensus.cc:2258] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:43.497838 26392 tablet_service.cc:1467] Processing CreateTablet for tablet 75f78aa840e84b7a9d4725d91c52fd05 (DEFAULT_TABLE table=test-table [id=e63518e2f939443797adca9dc6d93ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:43.498426 26392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f78aa840e84b7a9d4725d91c52fd05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:43.498947 26645 ts_tablet_manager.cc:1905] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:43.499141 26647 ts_tablet_manager.cc:1905] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:43.499140 26532 tablet_service.cc:1467] Processing CreateTablet for tablet 75f78aa840e84b7a9d4725d91c52fd05 (DEFAULT_TABLE table=test-table [id=e63518e2f939443797adca9dc6d93ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:43.499577 26532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f78aa840e84b7a9d4725d91c52fd05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:43.505935 26645 ts_tablet_manager.cc:1918] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:02:43.506120 26645 log.cc:1196] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/19545c4172424eb49c6b5846963e7dae
I20260619 15:02:43.506629 26645 ts_tablet_manager.cc:1939] T 19545c4172424eb49c6b5846963e7dae P 56c2eabc75fd43349318a25de0dfca1f: Deleting consensus metadata
I20260619 15:02:43.506882 26646 tablet_replica.cc:330] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54: stopping tablet replica
I20260619 15:02:43.507427 26646 raft_consensus.cc:2229] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:02:43.508033 26646 raft_consensus.cc:2258] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:43.508677 26103 catalog_manager.cc:4841] TS 56c2eabc75fd43349318a25de0dfca1f (127.24.103.67:36837): tablet 19545c4172424eb49c6b5846963e7dae (table test-table [id=3cad265e871941da85090cee4349ae55]) successfully deleted
I20260619 15:02:43.510233 26646 ts_tablet_manager.cc:1905] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:43.512355 26647 ts_tablet_manager.cc:1918] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:02:43.512406 26254 tablet_service.cc:1467] Processing CreateTablet for tablet 75f78aa840e84b7a9d4725d91c52fd05 (DEFAULT_TABLE table=test-table [id=e63518e2f939443797adca9dc6d93ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:43.512674 26647 log.cc:1196] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/19545c4172424eb49c6b5846963e7dae
I20260619 15:02:43.512924 26254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f78aa840e84b7a9d4725d91c52fd05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:43.513365 26647 ts_tablet_manager.cc:1939] T 19545c4172424eb49c6b5846963e7dae P d9390f6a2af746e78d4a654085261f6b: Deleting consensus metadata
I20260619 15:02:43.515383 26102 catalog_manager.cc:4841] TS d9390f6a2af746e78d4a654085261f6b (127.24.103.65:37253): tablet 19545c4172424eb49c6b5846963e7dae (table test-table [id=3cad265e871941da85090cee4349ae55]) successfully deleted
I20260619 15:02:43.517431 26621 tablet_bootstrap.cc:492] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54: Bootstrap starting.
I20260619 15:02:43.517439 26622 tablet_bootstrap.cc:492] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f: Bootstrap starting.
I20260619 15:02:43.519126 26622 tablet_bootstrap.cc:654] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:43.519126 26621 tablet_bootstrap.cc:654] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:43.520992 26624 tablet_bootstrap.cc:492] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b: Bootstrap starting.
I20260619 15:02:43.522600 26624 tablet_bootstrap.cc:654] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:43.535547 26624 tablet_bootstrap.cc:492] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b: No bootstrap required, opened a new log
I20260619 15:02:43.535838 26624 ts_tablet_manager.cc:1397] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260619 15:02:43.536779 26624 raft_consensus.cc:348] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.537170 26624 raft_consensus.cc:374] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:43.537322 26624 raft_consensus.cc:729] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9390f6a2af746e78d4a654085261f6b, State: Initialized, Role: FOLLOWER
I20260619 15:02:43.537680 26624 consensus_queue.cc:260] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.538414 26624 ts_tablet_manager.cc:1428] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 15:02:43.540244 26621 tablet_bootstrap.cc:492] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54: No bootstrap required, opened a new log
I20260619 15:02:43.540503 26621 ts_tablet_manager.cc:1397] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.000s
I20260619 15:02:43.541411 26621 raft_consensus.cc:348] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.541718 26621 raft_consensus.cc:374] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:43.541851 26621 raft_consensus.cc:729] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3d79bc059d42afad732c04c48e9a54, State: Initialized, Role: FOLLOWER
I20260619 15:02:43.542191 26621 consensus_queue.cc:260] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.542873 26621 ts_tablet_manager.cc:1428] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 15:02:43.545341 26646 ts_tablet_manager.cc:1918] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:02:43.545588 26646 log.cc:1196] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/19545c4172424eb49c6b5846963e7dae
I20260619 15:02:43.546260 26646 ts_tablet_manager.cc:1939] T 19545c4172424eb49c6b5846963e7dae P ea3d79bc059d42afad732c04c48e9a54: Deleting consensus metadata
I20260619 15:02:43.548544 26102 catalog_manager.cc:4841] TS ea3d79bc059d42afad732c04c48e9a54 (127.24.103.66:41111): tablet 19545c4172424eb49c6b5846963e7dae (table test-table [id=3cad265e871941da85090cee4349ae55]) successfully deleted
I20260619 15:02:43.549522 26622 tablet_bootstrap.cc:492] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f: No bootstrap required, opened a new log
I20260619 15:02:43.550179 26622 ts_tablet_manager.cc:1397] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.015s
I20260619 15:02:43.551074 26622 raft_consensus.cc:348] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.551280 26622 raft_consensus.cc:374] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:43.551405 26622 raft_consensus.cc:729] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c2eabc75fd43349318a25de0dfca1f, State: Initialized, Role: FOLLOWER
I20260619 15:02:43.551712 26622 consensus_queue.cc:260] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.552217 26622 ts_tablet_manager.cc:1428] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 15:02:43.658213 26632 raft_consensus.cc:482] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:02:43.658545 26632 raft_consensus.cc:504] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.659376 26632 leader_election.cc:288] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea3d79bc059d42afad732c04c48e9a54 (127.24.103.66:41111), 56c2eabc75fd43349318a25de0dfca1f (127.24.103.67:36837)
I20260619 15:02:43.660032 26413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f78aa840e84b7a9d4725d91c52fd05" candidate_uuid: "d9390f6a2af746e78d4a654085261f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d79bc059d42afad732c04c48e9a54" is_pre_election: true
I20260619 15:02:43.660111 26552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f78aa840e84b7a9d4725d91c52fd05" candidate_uuid: "d9390f6a2af746e78d4a654085261f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c2eabc75fd43349318a25de0dfca1f" is_pre_election: true
I20260619 15:02:43.660333 26413 raft_consensus.cc:2454] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9390f6a2af746e78d4a654085261f6b in term 0.
I20260619 15:02:43.660377 26552 raft_consensus.cc:2454] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9390f6a2af746e78d4a654085261f6b in term 0.
I20260619 15:02:43.661065 26206 leader_election.cc:302] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c2eabc75fd43349318a25de0dfca1f, d9390f6a2af746e78d4a654085261f6b; no voters:
I20260619 15:02:43.661418 26632 raft_consensus.cc:2781] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:02:43.661525 26632 raft_consensus.cc:482] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:02:43.661628 26632 raft_consensus.cc:3037] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:43.664876 26632 raft_consensus.cc:504] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.665599 26632 leader_election.cc:288] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [CANDIDATE]: Term 1 election: Requested vote from peers ea3d79bc059d42afad732c04c48e9a54 (127.24.103.66:41111), 56c2eabc75fd43349318a25de0dfca1f (127.24.103.67:36837)
I20260619 15:02:43.666146 26413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f78aa840e84b7a9d4725d91c52fd05" candidate_uuid: "d9390f6a2af746e78d4a654085261f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d79bc059d42afad732c04c48e9a54"
I20260619 15:02:43.666162 26552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f78aa840e84b7a9d4725d91c52fd05" candidate_uuid: "d9390f6a2af746e78d4a654085261f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c2eabc75fd43349318a25de0dfca1f"
I20260619 15:02:43.666394 26552 raft_consensus.cc:3037] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:43.666401 26413 raft_consensus.cc:3037] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:43.668911 26413 raft_consensus.cc:2454] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9390f6a2af746e78d4a654085261f6b in term 1.
I20260619 15:02:43.669361 26552 raft_consensus.cc:2454] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9390f6a2af746e78d4a654085261f6b in term 1.
I20260619 15:02:43.669438 26205 leader_election.cc:302] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9390f6a2af746e78d4a654085261f6b, ea3d79bc059d42afad732c04c48e9a54; no voters:
I20260619 15:02:43.669776 26632 raft_consensus.cc:2781] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:43.669968 26632 raft_consensus.cc:686] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 1 LEADER]: Becoming Leader. State: Replica: d9390f6a2af746e78d4a654085261f6b, State: Running, Role: LEADER
I20260619 15:02:43.670256 26632 consensus_queue.cc:237] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.673851 26116 catalog_manager.cc:5495] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b reported cstate change: term changed from 0 to 1, leader changed from <none> to d9390f6a2af746e78d4a654085261f6b (127.24.103.65). New cstate: current_term: 1 leader_uuid: "d9390f6a2af746e78d4a654085261f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
Jun 19 15:02:43 dist-test-slave-zpbw krb5kdc[26072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881361, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:43.760285 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:43.737859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45433 (local address 127.0.0.1:54388)
0619 15:02:43.738682 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:43.738698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:43.738935 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:43.739363 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:43.739366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:43.740143 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:43.740460 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:43.740469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.742247 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.742251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:43.743272 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:43.743284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.744889 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.744898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:43.745119 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:43.749744 (+ 4625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:43.749785 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:43.753184 (+ 3399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:43.756947 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:43.756963 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:43.756972 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:43.757607 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:43.757968 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:43.757973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:43.757977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:43.758189 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:43.759002 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:43.759013 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:43.759219 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 15:02:43.759413 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 15:02:43.759742 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":73,"thread_start_us":522,"threads_started":1}
I20260619 15:02:43.760615 26657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:43.737785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.24.103.126:45433)
0619 15:02:43.738535 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:43.738542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:43.738859 (+ 317us) server_negotiation.cc:407] Connection header received
0619 15:02:43.739075 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:43.739082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:43.739202 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:43.739344 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:43.740655 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.742049 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.743433 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.744288 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.745020 (+ 732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:43.753506 (+ 8486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:43.753512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:43.753516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:43.753554 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:43.756769 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:43.757745 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:43.757749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:43.757751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:43.757860 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:43.758407 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:43.758414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:43.758420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:43.758844 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:43.758991 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:43.759534 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 15:02:43.759940 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":570,"thread_start_us":391,"threads_started":1}
I20260619 15:02:43.765933 26116 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:54388:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 15:02:43.767405 26116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:02:43.774812 26532 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8e9ff5e46c49579620c207c6e1767c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be02f0acdfe24501870186e053e1db1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 15:02:43.774744 26392 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8e9ff5e46c49579620c207c6e1767c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be02f0acdfe24501870186e053e1db1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 15:02:43.774718 26254 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8e9ff5e46c49579620c207c6e1767c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be02f0acdfe24501870186e053e1db1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 15:02:43.775364 26392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8e9ff5e46c49579620c207c6e1767c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:43.775388 26532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8e9ff5e46c49579620c207c6e1767c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:43.775393 26254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8e9ff5e46c49579620c207c6e1767c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:43.783434 26622 tablet_bootstrap.cc:492] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f: Bootstrap starting.
I20260619 15:02:43.784180 26624 tablet_bootstrap.cc:492] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: Bootstrap starting.
I20260619 15:02:43.785620 26622 tablet_bootstrap.cc:654] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:43.785876 26624 tablet_bootstrap.cc:654] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:43.787429 26621 tablet_bootstrap.cc:492] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54: Bootstrap starting.
I20260619 15:02:43.788842 26622 tablet_bootstrap.cc:492] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f: No bootstrap required, opened a new log
I20260619 15:02:43.789000 26622 ts_tablet_manager.cc:1397] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260619 15:02:43.789119 26621 tablet_bootstrap.cc:654] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:43.790335 26622 raft_consensus.cc:348] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.790598 26622 raft_consensus.cc:374] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:43.790700 26622 raft_consensus.cc:729] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c2eabc75fd43349318a25de0dfca1f, State: Initialized, Role: FOLLOWER
I20260619 15:02:43.791249 26622 consensus_queue.cc:260] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.791893 26628 tablet_replica.cc:439] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } }
I20260619 15:02:43.792323 26628 tablet_replica.cc:442] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:43.792928 26624 tablet_bootstrap.cc:492] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: No bootstrap required, opened a new log
I20260619 15:02:43.793105 26624 ts_tablet_manager.cc:1397] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20260619 15:02:43.793557 26622 ts_tablet_manager.cc:1428] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 15:02:43.793869 26624 raft_consensus.cc:348] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.794112 26624 raft_consensus.cc:374] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:43.794198 26624 raft_consensus.cc:729] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9390f6a2af746e78d4a654085261f6b, State: Initialized, Role: FOLLOWER
I20260619 15:02:43.794442 26624 consensus_queue.cc:260] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.794943 26632 tablet_replica.cc:439] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } }
I20260619 15:02:43.795136 26624 ts_tablet_manager.cc:1428] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 15:02:43.795231 26632 tablet_replica.cc:442] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:43.795589 26621 tablet_bootstrap.cc:492] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54: No bootstrap required, opened a new log
I20260619 15:02:43.795750 26621 ts_tablet_manager.cc:1397] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 15:02:43.796844 26621 raft_consensus.cc:348] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.797166 26621 raft_consensus.cc:374] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:43.797318 26621 raft_consensus.cc:729] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3d79bc059d42afad732c04c48e9a54, State: Initialized, Role: FOLLOWER
I20260619 15:02:43.797981 26621 consensus_queue.cc:260] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.798506 26627 tablet_replica.cc:439] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } }
I20260619 15:02:43.799358 26627 tablet_replica.cc:442] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:43.799839 26621 ts_tablet_manager.cc:1428] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 15:02:43.883261 26632 raft_consensus.cc:482] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:02:43.883488 26632 raft_consensus.cc:504] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.884684 26632 leader_election.cc:288] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea3d79bc059d42afad732c04c48e9a54 (127.24.103.66:41111), 56c2eabc75fd43349318a25de0dfca1f (127.24.103.67:36837)
I20260619 15:02:43.884738 26552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8e9ff5e46c49579620c207c6e1767c" candidate_uuid: "d9390f6a2af746e78d4a654085261f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c2eabc75fd43349318a25de0dfca1f" is_pre_election: true
I20260619 15:02:43.884801 26413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8e9ff5e46c49579620c207c6e1767c" candidate_uuid: "d9390f6a2af746e78d4a654085261f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d79bc059d42afad732c04c48e9a54" is_pre_election: true
I20260619 15:02:43.884995 26552 raft_consensus.cc:2454] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9390f6a2af746e78d4a654085261f6b in term 0.
I20260619 15:02:43.885070 26413 raft_consensus.cc:2454] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9390f6a2af746e78d4a654085261f6b in term 0.
I20260619 15:02:43.885520 26205 leader_election.cc:302] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 56c2eabc75fd43349318a25de0dfca1f, d9390f6a2af746e78d4a654085261f6b, ea3d79bc059d42afad732c04c48e9a54; no voters:
I20260619 15:02:43.885847 26632 raft_consensus.cc:2781] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:02:43.885979 26632 raft_consensus.cc:482] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:02:43.886067 26632 raft_consensus.cc:3037] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:43.889421 26632 raft_consensus.cc:504] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.890635 26552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8e9ff5e46c49579620c207c6e1767c" candidate_uuid: "d9390f6a2af746e78d4a654085261f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c2eabc75fd43349318a25de0dfca1f"
I20260619 15:02:43.890719 26413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8e9ff5e46c49579620c207c6e1767c" candidate_uuid: "d9390f6a2af746e78d4a654085261f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3d79bc059d42afad732c04c48e9a54"
I20260619 15:02:43.890901 26552 raft_consensus.cc:3037] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:43.891004 26413 raft_consensus.cc:3037] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:43.893884 26552 raft_consensus.cc:2454] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9390f6a2af746e78d4a654085261f6b in term 1.
I20260619 15:02:43.894423 26413 raft_consensus.cc:2454] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9390f6a2af746e78d4a654085261f6b in term 1.
I20260619 15:02:43.894536 26206 leader_election.cc:302] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c2eabc75fd43349318a25de0dfca1f, d9390f6a2af746e78d4a654085261f6b; no voters:
I20260619 15:02:43.894934 26640 raft_consensus.cc:2781] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:43.895155 26640 raft_consensus.cc:686] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [term 1 LEADER]: Becoming Leader. State: Replica: d9390f6a2af746e78d4a654085261f6b, State: Running, Role: LEADER
I20260619 15:02:43.895503 26640 consensus_queue.cc:237] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } }
I20260619 15:02:43.895792 26632 leader_election.cc:288] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [CANDIDATE]: Term 1 election: Requested vote from peers ea3d79bc059d42afad732c04c48e9a54 (127.24.103.66:41111), 56c2eabc75fd43349318a25de0dfca1f (127.24.103.67:36837)
I20260619 15:02:43.897095 26632 tablet_replica.cc:439] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: TxnStatusTablet state changed. Reason: New leader d9390f6a2af746e78d4a654085261f6b. Latest consensus state: current_term: 1 leader_uuid: "d9390f6a2af746e78d4a654085261f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } }
I20260619 15:02:43.897442 26632 tablet_replica.cc:442] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: This TxnStatusTablet replica's current role is: LEADER
I20260619 15:02:43.898887 26116 catalog_manager.cc:5495] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b reported cstate change: term changed from 0 to 1, leader changed from <none> to d9390f6a2af746e78d4a654085261f6b (127.24.103.65). New cstate: current_term: 1 leader_uuid: "d9390f6a2af746e78d4a654085261f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260619 15:02:43.899325 26663 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 15:02:43.899677 26663 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 15:02:43.924865 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:43.918982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.24.103.65:37253)
0619 15:02:43.919171 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:43.919175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:43.919216 (+ 41us) server_negotiation.cc:407] Connection header received
0619 15:02:43.919556 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:43.919559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:43.919635 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:43.919725 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:02:43.920182 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.921013 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.922998 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.923418 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.923517 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:43.924162 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:02:43.924296 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:02:43.924361 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:43.924545 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 15:02:43.924613 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 15:02:43.925066 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:43.918850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37253 (local address 127.0.0.1:52876)
0619 15:02:43.919129 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:43.919154 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:43.919326 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:43.919785 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:43.919790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:43.919805 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 15:02:43.920016 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:43.920031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.921150 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.921159 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:43.922832 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:43.922850 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.923645 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.923652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:43.923878 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:43.923964 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 15:02:43.924397 (+ 433us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 15:02:43.924422 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 15:02:43.924532 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 15:02:43.924707 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 15:02:43.929301 26552 raft_consensus.cc:1261] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f [term 1 FOLLOWER]: Refusing update from remote peer d9390f6a2af746e78d4a654085261f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:43.929473 26413 raft_consensus.cc:1261] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54 [term 1 FOLLOWER]: Refusing update from remote peer d9390f6a2af746e78d4a654085261f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:43.930178 26632 consensus_queue.cc:1035] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:43.930430 26640 consensus_queue.cc:1035] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:43.935789 26627 tablet_replica.cc:439] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54: TxnStatusTablet state changed. Reason: New leader d9390f6a2af746e78d4a654085261f6b. Latest consensus state: current_term: 1 leader_uuid: "d9390f6a2af746e78d4a654085261f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } }
I20260619 15:02:43.936172 26627 tablet_replica.cc:442] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:43.937265 26628 tablet_replica.cc:439] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f: TxnStatusTablet state changed. Reason: New leader d9390f6a2af746e78d4a654085261f6b. Latest consensus state: current_term: 1 leader_uuid: "d9390f6a2af746e78d4a654085261f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } }
I20260619 15:02:43.937527 26628 tablet_replica.cc:442] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:43.939240 26640 tablet_replica.cc:439] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9390f6a2af746e78d4a654085261f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } }
I20260619 15:02:43.939558 26640 tablet_replica.cc:442] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: This TxnStatusTablet replica's current role is: LEADER
I20260619 15:02:43.941668 26629 tablet_replica.cc:439] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9390f6a2af746e78d4a654085261f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } }
I20260619 15:02:43.941946 26629 tablet_replica.cc:442] T 8b8e9ff5e46c49579620c207c6e1767c P d9390f6a2af746e78d4a654085261f6b: This TxnStatusTablet replica's current role is: LEADER
I20260619 15:02:43.941929 26636 mvcc.cc:205] Tried to move back new op lower bound from 7298586066647900160 to 7298586066518958080. Current Snapshot: MvccSnapshot[applied={T|T < 7298586066647900160}]
I20260619 15:02:43.942469 26628 tablet_replica.cc:439] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9390f6a2af746e78d4a654085261f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } }
I20260619 15:02:43.942699 26628 tablet_replica.cc:442] T 8b8e9ff5e46c49579620c207c6e1767c P 56c2eabc75fd43349318a25de0dfca1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:43.943822 26635 mvcc.cc:205] Tried to move back new op lower bound from 7298586066647900160 to 7298586066518958080. Current Snapshot: MvccSnapshot[applied={T|T < 7298586066647900160 or (T in {7298586066647900160})}]
I20260619 15:02:43.946152 26627 tablet_replica.cc:439] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9390f6a2af746e78d4a654085261f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9390f6a2af746e78d4a654085261f6b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37253 } } peers { permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 } } peers { permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 } } }
I20260619 15:02:43.946388 26627 tablet_replica.cc:442] T 8b8e9ff5e46c49579620c207c6e1767c P ea3d79bc059d42afad732c04c48e9a54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:43.966843 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:43.960936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:40831 (local address 127.24.103.65:37253)
0619 15:02:43.961160 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:43.961169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:43.961511 (+ 342us) server_negotiation.cc:407] Connection header received
0619 15:02:43.961737 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:43.961748 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:43.961801 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:43.961905 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:02:43.962416 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.963247 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.965088 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.965513 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.965675 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:43.966083 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:02:43.966191 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:02:43.966289 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:43.966417 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 15:02:43.966482 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 15:02:43.966964 26669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:43.960835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37253 (local address 127.24.103.65:40831)
0619 15:02:43.961415 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:43.961433 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:43.961618 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:43.961915 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:43.961919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:43.961928 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 15:02:43.962257 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:43.962265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.963378 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.963385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:43.964935 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:43.964947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:43.965715 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:43.965722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:43.965843 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:43.965896 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 15:02:43.966287 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 15:02:43.966298 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 15:02:43.966392 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 15:02:43.966539 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":326,"threads_started":1}
I20260619 15:02:43.977392 26413 raft_consensus.cc:1261] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54 [term 1 FOLLOWER]: Refusing update from remote peer d9390f6a2af746e78d4a654085261f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:43.977437 26552 raft_consensus.cc:1261] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f [term 1 FOLLOWER]: Refusing update from remote peer d9390f6a2af746e78d4a654085261f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:43.977998 26640 consensus_queue.cc:1035] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c2eabc75fd43349318a25de0dfca1f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:43.978495 26632 consensus_queue.cc:1035] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3d79bc059d42afad732c04c48e9a54" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:44.054329 26116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:02:44.054687 26116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:02:44.058025 26116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66ad46b02f4e402c821e646e09d09035: Sending DeleteTablet for 3 replicas of tablet 75f78aa840e84b7a9d4725d91c52fd05
I20260619 15:02:44.058981 26532 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f78aa840e84b7a9d4725d91c52fd05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:44 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:52986
I20260619 15:02:44.059062 26392 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f78aa840e84b7a9d4725d91c52fd05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:44 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:41690
I20260619 15:02:44.059914 26254 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f78aa840e84b7a9d4725d91c52fd05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:44 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:52858
I20260619 15:02:44.071363 26674 tablet_replica.cc:330] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f: stopping tablet replica
I20260619 15:02:44.071846 26674 raft_consensus.cc:2229] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:02:44.072211 26674 raft_consensus.cc:2258] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:44.073616 26674 ts_tablet_manager.cc:1905] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:44.073843 26675 tablet_replica.cc:330] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54: stopping tablet replica
I20260619 15:02:44.074113 26675 raft_consensus.cc:2229] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:02:44.074406 26675 raft_consensus.cc:2258] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:44.074710 26676 tablet_replica.cc:330] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b: stopping tablet replica
I20260619 15:02:44.075004 26676 raft_consensus.cc:2229] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:02:44.075506 26676 raft_consensus.cc:2258] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:44.076663 26675 ts_tablet_manager.cc:1905] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:44.077208 26676 ts_tablet_manager.cc:1905] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:44.081550 26674 ts_tablet_manager.cc:1918] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:02:44.081815 26674 log.cc:1196] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/75f78aa840e84b7a9d4725d91c52fd05
I20260619 15:02:44.082404 26674 ts_tablet_manager.cc:1939] T 75f78aa840e84b7a9d4725d91c52fd05 P 56c2eabc75fd43349318a25de0dfca1f: Deleting consensus metadata
I20260619 15:02:44.083840 26103 catalog_manager.cc:4841] TS 56c2eabc75fd43349318a25de0dfca1f (127.24.103.67:36837): tablet 75f78aa840e84b7a9d4725d91c52fd05 (table test-table [id=e63518e2f939443797adca9dc6d93ada]) successfully deleted
I20260619 15:02:44.083953 26675 ts_tablet_manager.cc:1918] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:02:44.084077 26675 log.cc:1196] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/75f78aa840e84b7a9d4725d91c52fd05
I20260619 15:02:44.084507 26675 ts_tablet_manager.cc:1939] T 75f78aa840e84b7a9d4725d91c52fd05 P ea3d79bc059d42afad732c04c48e9a54: Deleting consensus metadata
I20260619 15:02:44.085796 26102 catalog_manager.cc:4841] TS ea3d79bc059d42afad732c04c48e9a54 (127.24.103.66:41111): tablet 75f78aa840e84b7a9d4725d91c52fd05 (table test-table [id=e63518e2f939443797adca9dc6d93ada]) successfully deleted
Jun 19 15:02:44 dist-test-slave-zpbw krb5kdc[26072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:02:44.090977 26676 ts_tablet_manager.cc:1918] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:02:44.091157 26676 log.cc:1196] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/75f78aa840e84b7a9d4725d91c52fd05
I20260619 15:02:44.091708 26676 ts_tablet_manager.cc:1939] T 75f78aa840e84b7a9d4725d91c52fd05 P d9390f6a2af746e78d4a654085261f6b: Deleting consensus metadata
I20260619 15:02:44.092916 26102 catalog_manager.cc:4841] TS d9390f6a2af746e78d4a654085261f6b (127.24.103.65:37253): tablet 75f78aa840e84b7a9d4725d91c52fd05 (table test-table [id=e63518e2f939443797adca9dc6d93ada]) successfully deleted
I20260619 15:02:44.095981 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:44.069156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.24.103.65:37253)
0619 15:02:44.069408 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:44.069418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:44.073508 (+ 4090us) server_negotiation.cc:407] Connection header received
0619 15:02:44.073856 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:44.073864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:44.073989 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:44.074072 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:44.077434 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:44.078562 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:44.082620 (+ 4058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:44.083408 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:44.083523 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:44.088585 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:44.088599 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:44.088617 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:44.088660 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:44.092364 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:44.093226 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:44.093241 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:44.093244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:44.093373 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:44.093751 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:44.093756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:44.093759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:44.094121 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:44.094449 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:44.094818 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 15:02:44.095061 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 15:02:44.096825 26319 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52880
I20260619 15:02:44.122025 26657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:44.106337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.24.103.126:45433)
0619 15:02:44.106637 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:44.106648 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:44.107165 (+ 517us) server_negotiation.cc:407] Connection header received
0619 15:02:44.107355 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:44.107361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:44.107447 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:44.107568 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:44.108929 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:44.110061 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:44.111478 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:44.112178 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:44.112305 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:44.115620 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:44.115627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:44.115633 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:44.115677 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:44.119271 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:44.119994 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:44.120000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:44.120004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:44.120155 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:44.120525 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:44.120530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:44.120534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:44.120902 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:44.121145 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:44.121434 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 15:02:44.121678 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260619 15:02:44.122692 26116 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54390
I20260619 15:02:44.124737 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 26188
I20260619 15:02:44.140275 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:44.126098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52886 (local address 127.24.103.65:37253)
0619 15:02:44.126328 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:44.126337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:44.126794 (+ 457us) server_negotiation.cc:407] Connection header received
0619 15:02:44.126990 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:44.126997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:44.127081 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:44.127187 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:44.128238 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:44.129407 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:44.130838 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:44.131528 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:44.131660 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:44.133928 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:44.133935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:44.133940 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:44.133988 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:44.137559 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:44.138234 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:44.138240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:44.138243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:44.138398 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:44.138764 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:44.138770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:44.138802 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:44.139166 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:44.139436 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:44.139684 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 15:02:44.139912 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 15:02:44.140852 26319 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52886
W20260619 15:02:44.141422 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 26188: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:02:44.141625 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 26188
I20260619 15:02:44.159854 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 26328
Jun 19 15:02:44 dist-test-slave-zpbw krb5kdc[26072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:02:44.174971 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:44.160573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.24.103.66:41111)
0619 15:02:44.161122 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:44.161126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:44.161143 (+ 17us) server_negotiation.cc:407] Connection header received
0619 15:02:44.161197 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:44.161201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:44.161258 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:44.161352 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:44.162566 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:44.163391 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:44.164664 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:44.165068 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:44.165168 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:44.169515 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:44.169531 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:44.169547 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:44.169595 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:44.172494 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:44.173141 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:44.173148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:44.173150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:44.173282 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:44.173618 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:44.173621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:44.173623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:44.173967 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:44.174164 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:44.174414 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 15:02:44.174624 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":315,"threads_started":1}
W20260619 15:02:44.175693 26458 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41694
W20260619 15:02:44.176311 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 26328: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:02:44.176463 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 26328
I20260619 15:02:44.192821 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 26467
Jun 19 15:02:44 dist-test-slave-zpbw krb5kdc[26072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:02:44.208600 26683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:44.193693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.24.103.67:36837)
0619 15:02:44.194165 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:44.194170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:44.194186 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:02:44.194240 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:44.194244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:44.194307 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:44.194396 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:44.195807 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:44.196707 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:44.197902 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:44.198298 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:44.198398 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:44.202644 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:44.202659 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:44.202675 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:44.202709 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:44.205800 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:44.206515 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:44.206523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:44.206525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:44.206662 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:44.207137 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:44.207142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:44.207144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:44.207481 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:44.207691 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:44.208009 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 15:02:44.208198 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":242,"threads_started":1}
W20260619 15:02:44.209278 26597 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52994
W20260619 15:02:44.210016 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 26467: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:02:44.210157 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 26467
I20260619 15:02:44.226954 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 26085
I20260619 15:02:44.228237 26157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:02:44.343649 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 26085
2026-06-19T15:02:44Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4811 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:02:44 dist-test-slave-zpbw krb5kdc[26687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:02:44 dist-test-slave-zpbw krb5kdc[26687](info): set up 2 sockets
Jun 19 15:02:44 dist-test-slave-zpbw krb5kdc[26687](info): commencing operation
krb5kdc: starting...
W20260619 15:02:45.427047 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:02:45 dist-test-slave-zpbw krb5kdc[26687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:02:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:02:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:02:45.618010 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35505
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:35505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:45.940913 26700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:45.941219 26700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:45.941320 26700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:45.948720 26700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:02:45.948841 26700 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 15:02:45.948884 26700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:45.948945 26700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:45.948997 26700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:02:45.949061 26700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:02:45.959007 26700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:35505
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:45.960350 26700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:45.961619 26700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:45.973721 26706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:45.973649 26708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:45.973757 26705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:45.974928 26700 server_base.cc:1034] running on GCE node
I20260619 15:02:45.975703 26700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:45.977054 26700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:45.978302 26700 hybrid_clock.cc:648] HybridClock initialized: now 1781881365978253 us; error 38 us; skew 500 ppm
I20260619 15:02:45.978663 26700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:45 dist-test-slave-zpbw krb5kdc[26687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881365, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:45.981817 26700 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:02:45.983449 26700 webserver.cc:460] Webserver started at http://127.24.103.126:33315/ using document root <none> and password file <none>
I20260619 15:02:45.984160 26700 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:45.984290 26700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:45.984618 26700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:45.987620 26700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "1ae4cac1dd264b61b24758a57db42f5e"
format_stamp: "Formatted at 2026-06-19 15:02:45 on dist-test-slave-zpbw"
server_key: "372499a4125785970b3f64a38032b630"
server_key_iv: "a2ad68946f2783bff1af0119746e0579"
server_key_version: "encryptionkey@0"
I20260619 15:02:45.988392 26700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "1ae4cac1dd264b61b24758a57db42f5e"
format_stamp: "Formatted at 2026-06-19 15:02:45 on dist-test-slave-zpbw"
server_key: "372499a4125785970b3f64a38032b630"
server_key_iv: "a2ad68946f2783bff1af0119746e0579"
server_key_version: "encryptionkey@0"
I20260619 15:02:45.993592 26700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:02:45.997625 26715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:45.999099 26700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:02:45.999290 26700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "1ae4cac1dd264b61b24758a57db42f5e"
format_stamp: "Formatted at 2026-06-19 15:02:45 on dist-test-slave-zpbw"
server_key: "372499a4125785970b3f64a38032b630"
server_key_iv: "a2ad68946f2783bff1af0119746e0579"
server_key_version: "encryptionkey@0"
I20260619 15:02:45.999540 26700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:46.035773 26700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:46.040441 26700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:46.040822 26700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:46.064906 26700 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:35505
I20260619 15:02:46.064932 26776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:35505 every 8 connection(s)
I20260619 15:02:46.066466 26700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:02:46.070499 26777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:46.077103 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 26700
I20260619 15:02:46.077337 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:02:46.078174 24989 external_mini_cluster.cc:1428] Setting key 1d0eb38e387dafbd21154e89aa189c1a
I20260619 15:02:46.082233 26777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e: Bootstrap starting.
I20260619 15:02:46.085906 26777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:46.087204 26777 log.cc:824] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:46.090237 26777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e: No bootstrap required, opened a new log
Jun 19 15:02:46 dist-test-slave-zpbw krb5kdc[26687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:46.100981 26777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae4cac1dd264b61b24758a57db42f5e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35505 } }
I20260619 15:02:46.101531 26777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:46.101646 26777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae4cac1dd264b61b24758a57db42f5e, State: Initialized, Role: FOLLOWER
I20260619 15:02:46.102381 26777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae4cac1dd264b61b24758a57db42f5e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35505 } }
I20260619 15:02:46.102664 26777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:02:46.102823 26777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:02:46.103012 26777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:46.104127 26780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:46.080523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.24.103.126:35505)
0619 15:02:46.081380 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:46.081392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:46.081435 (+ 43us) server_negotiation.cc:407] Connection header received
0619 15:02:46.083438 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:46.083461 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:46.083906 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:46.084257 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:46.086133 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:46.087767 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:46.089353 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:46.090031 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:46.090167 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:46.094962 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:46.094975 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:46.095001 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:46.095061 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:46.099171 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:46.099971 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:46.099980 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:46.099989 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:46.100215 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:46.100565 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:46.100569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:46.100572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:46.101292 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:46.101643 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:46.102138 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 15:02:46.102603 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":400,"threads_started":1}
I20260619 15:02:46.106012 26777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae4cac1dd264b61b24758a57db42f5e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35505 } }
I20260619 15:02:46.106619 26777 leader_election.cc:302] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ae4cac1dd264b61b24758a57db42f5e; no voters:
I20260619 15:02:46.107369 26777 leader_election.cc:288] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:02:46.107487 26782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:46.108165 26782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [term 1 LEADER]: Becoming Leader. State: Replica: 1ae4cac1dd264b61b24758a57db42f5e, State: Running, Role: LEADER
I20260619 15:02:46.108691 26782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae4cac1dd264b61b24758a57db42f5e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35505 } }
I20260619 15:02:46.109566 26777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:02:46.111758 26784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ae4cac1dd264b61b24758a57db42f5e. Latest consensus state: current_term: 1 leader_uuid: "1ae4cac1dd264b61b24758a57db42f5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae4cac1dd264b61b24758a57db42f5e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35505 } } }
I20260619 15:02:46.112116 26784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [sys.catalog]: This master's current role is: LEADER
I20260619 15:02:46.113801 26783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ae4cac1dd264b61b24758a57db42f5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae4cac1dd264b61b24758a57db42f5e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35505 } } }
I20260619 15:02:46.114045 26790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:02:46.114168 26783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e [sys.catalog]: This master's current role is: LEADER
I20260619 15:02:46.120371 26790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:02:46.130400 26790 catalog_manager.cc:1347] Generated new cluster ID: b6e014df104c4db2b7b405f25aa64f40
I20260619 15:02:46.130548 26790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:02:46.142741 26790 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:02:46.144145 26790 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:02:46.151018 26790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e: Generated new TSK 0
I20260619 15:02:46.151827 26790 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:02:46.206614 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35505
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:46.551087 26803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:46.551399 26803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:46.551504 26803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:46.559258 26803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:46.559394 26803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:46.559482 26803 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 15:02:46.559536 26803 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 15:02:46.559609 26803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:02:46.569707 26803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:35505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:46.571107 26803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:46.572453 26803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:46.586653 26808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:46.586722 26811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:46.586719 26809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:46.587984 26803 server_base.cc:1034] running on GCE node
I20260619 15:02:46.588652 26803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:46.589573 26803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:46.590864 26803 hybrid_clock.cc:648] HybridClock initialized: now 1781881366590759 us; error 38 us; skew 500 ppm
I20260619 15:02:46.591250 26803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:46 dist-test-slave-zpbw krb5kdc[26687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881366, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:46.594620 26803 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:02:46.596290 26803 webserver.cc:460] Webserver started at http://127.24.103.65:41253/ using document root <none> and password file <none>
I20260619 15:02:46.597064 26803 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:46.597223 26803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:46.597561 26803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:46.600796 26803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "0b7b4bb20a254493a141175daf679534"
format_stamp: "Formatted at 2026-06-19 15:02:46 on dist-test-slave-zpbw"
server_key: "38502b2173ebd7eec709d6be562f475f"
server_key_iv: "d0094d91690bfd540c1c9040fe3a1002"
server_key_version: "encryptionkey@0"
I20260619 15:02:46.601632 26803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "0b7b4bb20a254493a141175daf679534"
format_stamp: "Formatted at 2026-06-19 15:02:46 on dist-test-slave-zpbw"
server_key: "38502b2173ebd7eec709d6be562f475f"
server_key_iv: "d0094d91690bfd540c1c9040fe3a1002"
server_key_version: "encryptionkey@0"
I20260619 15:02:46.607586 26803 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 15:02:46.611461 26818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:46.612941 26803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:02:46.613164 26803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "0b7b4bb20a254493a141175daf679534"
format_stamp: "Formatted at 2026-06-19 15:02:46 on dist-test-slave-zpbw"
server_key: "38502b2173ebd7eec709d6be562f475f"
server_key_iv: "d0094d91690bfd540c1c9040fe3a1002"
server_key_version: "encryptionkey@0"
I20260619 15:02:46.613394 26803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:46.650449 26803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:46.655085 26803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:46.655414 26803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:46.658226 26803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:46.658376 26803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:46.658512 26803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:46.658615 26803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 15:02:46 dist-test-slave-zpbw krb5kdc[26687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881366, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:46.686426 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:46.659045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:46225 (local address 127.24.103.126:35505)
0619 15:02:46.659575 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:46.659582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:46.660838 (+ 1256us) server_negotiation.cc:407] Connection header received
0619 15:02:46.664021 (+ 3183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:46.664029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:46.664139 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:46.664268 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:46.667244 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:46.668119 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:46.671552 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:46.672209 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:46.672372 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:46.677745 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:46.677754 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:46.677761 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:46.677815 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:46.681313 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:46.682522 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:46.682530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:46.682537 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:46.682762 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:46.683568 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:46.683577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:46.683583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:46.684017 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:46.684201 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:46.685639 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0619 15:02:46.685946 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":278,"threads_started":1}
I20260619 15:02:46.688788 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:46.659680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35505 (local address 127.24.103.65:46225)
0619 15:02:46.660629 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:46.660690 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:46.663597 (+ 2907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:46.664618 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:46.664634 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:46.665769 (+ 1135us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:46.666935 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:46.666977 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:46.668316 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:46.668337 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:46.671300 (+ 2963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:46.671322 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:46.672503 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:46.672514 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:46.672926 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:46.673849 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:46.673898 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:46.677426 (+ 3528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:46.681518 (+ 4092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:46.681532 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:46.681565 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:46.682328 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:46.682984 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:46.682992 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:46.682999 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:46.683390 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:46.684288 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:46.684302 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:46.685059 (+ 757us) client_negotiation.cc:769] Sending connection context
0619 15:02:46.685480 (+ 421us) client_negotiation.cc:240] Negotiation successful
0619 15:02:46.686303 (+ 823us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":358,"threads_started":1}
I20260619 15:02:46.711364 26803 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:32797
I20260619 15:02:46.711458 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:46.695079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:49011 (local address 127.24.103.126:35505)
0619 15:02:46.695264 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:46.695269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:46.695884 (+ 615us) server_negotiation.cc:407] Connection header received
0619 15:02:46.696102 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:46.696108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:46.696184 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:46.696296 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:46.697233 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:46.697935 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:46.699813 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:46.700646 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:46.700808 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:46.703631 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:46.703641 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:46.703649 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:46.703700 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:46.707720 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:46.708947 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:46.708953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:46.708959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:46.709158 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:46.709580 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:46.709583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:46.709586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:46.709894 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:46.710067 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:46.710743 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 15:02:46.711053 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 15:02:46.711440 26938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:32797 every 8 connection(s)
I20260619 15:02:46.712467 26902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:46.695161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35505 (local address 127.24.103.65:49011)
0619 15:02:46.695776 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:46.695794 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:46.695962 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:46.696321 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:46.696326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:46.696799 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:46.697081 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:46.697089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:46.698080 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:46.698086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:46.699632 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:46.699650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:46.700891 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:46.700901 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:46.701134 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:46.701972 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:46.702010 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:46.703318 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:46.707927 (+ 4609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:46.707933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:46.707942 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:46.708751 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:46.709299 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:46.709304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:46.709307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:46.709457 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:46.710049 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:46.710058 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:46.710344 (+ 286us) client_negotiation.cc:769] Sending connection context
0619 15:02:46.711628 (+ 1284us) client_negotiation.cc:240] Negotiation successful
0619 15:02:46.712015 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":352,"threads_started":1}
I20260619 15:02:46.713053 26803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:02:46.715342 26939 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35505
I20260619 15:02:46.715592 26939 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:46.716342 26939 heartbeater.cc:511] Master 127.24.103.126:35505 requested a full tablet report, sending...
I20260619 15:02:46.718420 26732 ts_manager.cc:194] Registered new tserver with Master: 0b7b4bb20a254493a141175daf679534 (127.24.103.65:32797)
I20260619 15:02:46.719449 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 26803
I20260619 15:02:46.719651 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:02:46.720455 24989 external_mini_cluster.cc:1428] Setting key 127a010b59c1fdc4ed23fc947c056d75
I20260619 15:02:46.720661 26732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:46225
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:02:46.758164 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35505
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:47.086349 26943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:47.086704 26943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:47.086897 26943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:47.094301 26943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:47.094470 26943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:47.094633 26943 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 15:02:47.094725 26943 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 15:02:47.094882 26943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:02:47.104981 26943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:35505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:47.106333 26943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:47.107712 26943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:47.121392 26948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:47.121419 26951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:47.121464 26949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:47.122272 26943 server_base.cc:1034] running on GCE node
I20260619 15:02:47.122952 26943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:47.123893 26943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:47.125176 26943 hybrid_clock.cc:648] HybridClock initialized: now 1781881367125131 us; error 31 us; skew 500 ppm
I20260619 15:02:47.125569 26943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:47 dist-test-slave-zpbw krb5kdc[26687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881367, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:47.128880 26943 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:02:47.130374 26943 webserver.cc:460] Webserver started at http://127.24.103.66:38815/ using document root <none> and password file <none>
I20260619 15:02:47.131148 26943 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:47.131281 26943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:47.131601 26943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:47.134575 26943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "0e781bf4699d49288db3eb6c3c47ee62"
format_stamp: "Formatted at 2026-06-19 15:02:47 on dist-test-slave-zpbw"
server_key: "3fc6e025fdd676a939a46557737208b8"
server_key_iv: "408ff4c75413c59bcfe143e7928d3c8e"
server_key_version: "encryptionkey@0"
I20260619 15:02:47.135454 26943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "0e781bf4699d49288db3eb6c3c47ee62"
format_stamp: "Formatted at 2026-06-19 15:02:47 on dist-test-slave-zpbw"
server_key: "3fc6e025fdd676a939a46557737208b8"
server_key_iv: "408ff4c75413c59bcfe143e7928d3c8e"
server_key_version: "encryptionkey@0"
I20260619 15:02:47.140559 26943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:02:47.144085 26958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:47.145440 26943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:02:47.145675 26943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "0e781bf4699d49288db3eb6c3c47ee62"
format_stamp: "Formatted at 2026-06-19 15:02:47 on dist-test-slave-zpbw"
server_key: "3fc6e025fdd676a939a46557737208b8"
server_key_iv: "408ff4c75413c59bcfe143e7928d3c8e"
server_key_version: "encryptionkey@0"
I20260619 15:02:47.145884 26943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:47.166746 26943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:47.171371 26943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:47.171684 26943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:47.174337 26943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:47.174479 26943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:47.174616 26943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:47.174708 26943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 15:02:47 dist-test-slave-zpbw krb5kdc[26687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881367, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:47.200857 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.175191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:54095 (local address 127.24.103.126:35505)
0619 15:02:47.175412 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:47.175418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:47.177053 (+ 1635us) server_negotiation.cc:407] Connection header received
0619 15:02:47.179170 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:47.179176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:47.179267 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:47.179401 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:47.181863 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.183018 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.186518 (+ 3500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.187092 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.187191 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.193073 (+ 5882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:47.193078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:47.193082 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:47.193118 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:47.196819 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:47.198112 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:47.198116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:47.198118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:47.198245 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:47.198880 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:47.198885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:47.198888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:47.199160 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:47.199273 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:47.200426 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0619 15:02:47.200597 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 15:02:47.203229 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.175996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35505 (local address 127.24.103.66:54095)
0619 15:02:47.176869 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:47.176912 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:47.178894 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:47.179723 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:47.179739 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:47.180643 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:47.181658 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.181685 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.183243 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.183253 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.186310 (+ 3057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.186335 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.187369 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.187378 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.187762 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.188636 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:47.188684 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:47.192730 (+ 4046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:47.197074 (+ 4344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:47.197088 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:47.197117 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:47.197945 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:47.198375 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:47.198383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:47.198389 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:47.198719 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:47.199306 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:47.199319 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:47.199936 (+ 617us) client_negotiation.cc:769] Sending connection context
0619 15:02:47.200342 (+ 406us) client_negotiation.cc:240] Negotiation successful
0619 15:02:47.201083 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":374,"threads_started":1}
I20260619 15:02:47.230438 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.214212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:49521 (local address 127.24.103.126:35505)
0619 15:02:47.214479 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:47.214489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:47.214583 (+ 94us) server_negotiation.cc:407] Connection header received
0619 15:02:47.214873 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:47.214878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:47.215007 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:47.215117 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:47.216111 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.216986 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.219640 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.220195 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.220342 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.223109 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:47.223115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:47.223123 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:47.223165 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:47.226963 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:47.227825 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:47.227833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:47.227836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:47.228027 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:47.228583 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:47.228589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:47.228593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:47.229126 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:47.229290 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:47.229870 (+ 580us) server_negotiation.cc:299] Negotiation successful
0619 15:02:47.230089 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 15:02:47.230592 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.213825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35505 (local address 127.24.103.66:49521)
0619 15:02:47.214484 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:47.214505 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:47.214698 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:47.215139 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:47.215143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:47.215635 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:47.215952 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.215964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.217150 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.217159 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.219356 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.219378 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.220433 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.220443 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.220687 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.221423 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:47.221455 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:47.222868 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:47.227102 (+ 4234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:47.227106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:47.227109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:47.227647 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:47.228193 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:47.228201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:47.228207 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:47.228412 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:47.229302 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:47.229309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:47.229584 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 15:02:47.229756 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 15:02:47.230114 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":452,"threads_started":1}
I20260619 15:02:47.238188 26943 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:34843
I20260619 15:02:47.238199 27077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:34843 every 8 connection(s)
I20260619 15:02:47.239507 26943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:02:47.241467 27078 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35505
I20260619 15:02:47.241698 27078 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:47.242357 27078 heartbeater.cc:511] Master 127.24.103.126:35505 requested a full tablet report, sending...
I20260619 15:02:47.243842 26731 ts_manager.cc:194] Registered new tserver with Master: 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843)
I20260619 15:02:47.244714 26731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:54095
I20260619 15:02:47.248255 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 26943
I20260619 15:02:47.248402 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:02:47.248808 24989 external_mini_cluster.cc:1428] Setting key 15ecca0fd7fc5c83138e4f7d59582292
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:02:47.287935 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35505
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:47.616727 27082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:47.617064 27082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:47.617240 27082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:47.626094 27082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:47.626260 27082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:47.626405 27082 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 15:02:47.626492 27082 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 15:02:47.626614 27082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:02:47.636953 27082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:35505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:47.638329 27082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:47.639789 27082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:47.653157 27090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:47.653205 27087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:47.653299 27088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:47.654059 27082 server_base.cc:1034] running on GCE node
I20260619 15:02:47.654764 27082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:47.655841 27082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:47.657035 27082 hybrid_clock.cc:648] HybridClock initialized: now 1781881367656971 us; error 68 us; skew 500 ppm
I20260619 15:02:47.657356 27082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:47 dist-test-slave-zpbw krb5kdc[26687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881367, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:47.660756 27082 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:02:47.662271 27082 webserver.cc:460] Webserver started at http://127.24.103.67:36345/ using document root <none> and password file <none>
I20260619 15:02:47.663008 27082 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:47.663113 27082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:47.663367 27082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:47.666489 27082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "fec1d74847e4425181613c9e261b21be"
format_stamp: "Formatted at 2026-06-19 15:02:47 on dist-test-slave-zpbw"
server_key: "47e3a875cf34e17c52f706424ec32f91"
server_key_iv: "282d4b8e29170760febcbc3186ce0545"
server_key_version: "encryptionkey@0"
I20260619 15:02:47.667342 27082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "fec1d74847e4425181613c9e261b21be"
format_stamp: "Formatted at 2026-06-19 15:02:47 on dist-test-slave-zpbw"
server_key: "47e3a875cf34e17c52f706424ec32f91"
server_key_iv: "282d4b8e29170760febcbc3186ce0545"
server_key_version: "encryptionkey@0"
I20260619 15:02:47.672765 27082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 15:02:47.676527 27097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:47.677951 27082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 15:02:47.678162 27082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "fec1d74847e4425181613c9e261b21be"
format_stamp: "Formatted at 2026-06-19 15:02:47 on dist-test-slave-zpbw"
server_key: "47e3a875cf34e17c52f706424ec32f91"
server_key_iv: "282d4b8e29170760febcbc3186ce0545"
server_key_version: "encryptionkey@0"
I20260619 15:02:47.678398 27082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:47.695402 27082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:47.700001 27082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:47.700330 27082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:47.703138 27082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:47.703271 27082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:47.703400 27082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:47.703503 27082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 15:02:47 dist-test-slave-zpbw krb5kdc[26687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881367, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:47.723692 26939 heartbeater.cc:503] Master 127.24.103.126:35505 was elected leader, sending a full tablet report...
I20260619 15:02:47.734067 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.704038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:57209 (local address 127.24.103.126:35505)
0619 15:02:47.704272 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:47.704279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:47.705970 (+ 1691us) server_negotiation.cc:407] Connection header received
0619 15:02:47.708947 (+ 2977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:47.708954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:47.709045 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:47.709173 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:47.714896 (+ 5723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.716150 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.718885 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.719700 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.719884 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.725275 (+ 5391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:47.725285 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:47.725293 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:47.725346 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:47.729655 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:47.730883 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:47.730890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:47.730893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:47.731066 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:47.731717 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:47.731726 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:47.731732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:47.732237 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:47.732475 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:47.733447 (+ 972us) server_negotiation.cc:299] Negotiation successful
0619 15:02:47.733685 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 15:02:47.735656 27106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.704812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35505 (local address 127.24.103.67:57209)
0619 15:02:47.705775 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:47.705833 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:47.708635 (+ 2802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:47.709502 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:47.709516 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:47.710460 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:47.714552 (+ 4092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.714601 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.716485 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.716490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.718656 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.718678 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.720013 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.720022 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.720228 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.721161 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:47.721186 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:47.724829 (+ 3643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:47.729851 (+ 5022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:47.729865 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:47.729893 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:47.730643 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:47.731209 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:47.731214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:47.731218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:47.731524 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:47.732392 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:47.732412 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:47.732912 (+ 500us) client_negotiation.cc:769] Sending connection context
0619 15:02:47.733311 (+ 399us) client_negotiation.cc:240] Negotiation successful
0619 15:02:47.733929 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":404,"threads_started":1}
I20260619 15:02:47.752823 27082 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:35907
I20260619 15:02:47.752866 27216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:35907 every 8 connection(s)
I20260619 15:02:47.754271 27082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:02:47.756896 27217 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35505
I20260619 15:02:47.757182 27217 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:47.757293 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 27082
I20260619 15:02:47.757469 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:02:47.758023 24989 external_mini_cluster.cc:1428] Setting key 6dc9825fe51ecb5678dd2c6864e905bb
I20260619 15:02:47.758085 27217 heartbeater.cc:511] Master 127.24.103.126:35505 requested a full tablet report, sending...
I20260619 15:02:47.759738 26731 ts_manager.cc:194] Registered new tserver with Master: fec1d74847e4425181613c9e261b21be (127.24.103.67:35907)
I20260619 15:02:47.759948 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.743678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35505 (local address 127.24.103.67:37447)
0619 15:02:47.744324 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:47.744353 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:47.744527 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:47.744886 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:47.744890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:47.745702 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:47.746260 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.746280 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.748111 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.748120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.750629 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.750647 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.751533 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.751540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.751718 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.752484 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:47.752502 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:47.753353 (+ 851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:47.756857 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:47.756885 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:47.756889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:47.757534 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:47.757907 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:47.757912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:47.757916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:47.758048 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:47.758739 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:47.758745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:47.758974 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 15:02:47.759170 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 15:02:47.759506 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":333,"threads_started":1}
I20260619 15:02:47.760850 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.743589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:37447 (local address 127.24.103.126:35505)
0619 15:02:47.744018 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:47.744023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:47.744460 (+ 437us) server_negotiation.cc:407] Connection header received
0619 15:02:47.744664 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:47.744668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:47.744741 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:47.744824 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:47.746513 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.747874 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.750881 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.751314 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.751477 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.753563 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:47.753569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:47.753572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:47.753605 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:47.756735 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:47.757639 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:47.757645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:47.757649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:47.757789 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:47.758216 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:47.758220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:47.758222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:47.758586 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:47.758743 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:47.760225 (+ 1482us) server_negotiation.cc:299] Negotiation successful
0619 15:02:47.760494 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260619 15:02:47.760921 26731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:57209
I20260619 15:02:47.761365 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 15:02:47 dist-test-slave-zpbw krb5kdc[26687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 15:02:47 dist-test-slave-zpbw krb5kdc[26687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:47.809198 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.794684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44268 (local address 127.24.103.126:35505)
0619 15:02:47.794972 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:47.794978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:47.795192 (+ 214us) server_negotiation.cc:407] Connection header received
0619 15:02:47.795361 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:47.795365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:47.795443 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:47.795534 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:47.796970 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.797725 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.799055 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.799458 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.799592 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.803524 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:47.803530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:47.803534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:47.803601 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:47.806424 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:47.807217 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:47.807221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:47.807224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:47.807374 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:47.807721 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:47.807724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:47.807726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:47.808038 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:47.808236 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:47.808522 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 15:02:47.808724 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260619 15:02:47.813498 26731 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:02:47.820796 26731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:02:47.845949 27231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.832973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34843 (local address 127.0.0.1:42588)
0619 15:02:47.833970 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:47.834027 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:47.834243 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:47.835360 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:47.835373 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:47.835417 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:47.835954 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.835971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.838810 (+ 2839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.838820 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.842070 (+ 3250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.842091 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.844743 (+ 2652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.844753 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.845076 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.845159 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 15:02:47.845293 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 15:02:47.845526 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":1611,"threads_started":1}
I20260619 15:02:47.846743 27230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.833151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:32797 (local address 127.0.0.1:59840)
0619 15:02:47.834957 (+ 1806us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:47.834979 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:47.835118 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:47.836317 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:47.836326 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:47.836361 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:47.836908 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.836924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.839750 (+ 2826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.839760 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.842832 (+ 3072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.842853 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.845534 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.845586 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.845884 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.845943 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 15:02:47.846027 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 15:02:47.846226 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1654,"thread_start_us":368,"threads_started":1}
I20260619 15:02:47.847942 27232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.833408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.24.103.66:34843)
0619 15:02:47.834614 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:47.834623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:47.834655 (+ 32us) server_negotiation.cc:407] Connection header received
0619 15:02:47.834765 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:47.834798 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:47.835082 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:47.836223 (+ 1141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:47.836473 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.838575 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.842399 (+ 3824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.844408 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.846314 (+ 1906us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.846543 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:47.847586 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0619 15:02:47.847743 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":929,"threads_started":1}
I20260619 15:02:47.848296 27233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.833716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.24.103.65:32797)
0619 15:02:47.835645 (+ 1929us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:47.835654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:47.835682 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:02:47.835790 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:47.835797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:47.836052 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:47.837144 (+ 1092us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:47.837428 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.839551 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.843106 (+ 3555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.845203 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.847528 (+ 2325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.847792 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:47.847864 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 15:02:47.848034 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1767,"thread_start_us":2034,"threads_started":1}
I20260619 15:02:47.851876 26869 tablet_service.cc:1467] Processing CreateTablet for tablet abaf9caf88e44b7aa604b8429d98a36d (DEFAULT_TABLE table=test-table [id=a1147afc297a4223955cf8ccb8742be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:47.851883 27008 tablet_service.cc:1467] Processing CreateTablet for tablet abaf9caf88e44b7aa604b8429d98a36d (DEFAULT_TABLE table=test-table [id=a1147afc297a4223955cf8ccb8742be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:47.853226 27008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abaf9caf88e44b7aa604b8429d98a36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:47.853307 26869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abaf9caf88e44b7aa604b8429d98a36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:47.856953 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.835566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55002 (local address 127.24.103.67:35907)
0619 15:02:47.847992 (+ 12426us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:47.847999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:47.848031 (+ 32us) server_negotiation.cc:407] Connection header received
0619 15:02:47.848142 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:47.848150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:47.848440 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:47.849446 (+ 1006us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:47.850258 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.851873 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.854365 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.855834 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.855966 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.856188 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:47.856465 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 15:02:47.856616 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12242,"thread_start_us":12631,"threads_started":1}
I20260619 15:02:47.856979 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:47.834992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:35907 (local address 127.0.0.1:55002)
0619 15:02:47.845598 (+ 10606us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:47.845617 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:47.845747 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:47.849468 (+ 3721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:47.849473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:47.849504 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:47.849870 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.849883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.852032 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.852038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.854169 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:47.854183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:47.856057 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:47.856066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:47.856253 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:47.856292 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 15:02:47.856400 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 15:02:47.856546 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10439,"thread_start_us":10860,"threads_started":1}
I20260619 15:02:47.861223 27147 tablet_service.cc:1467] Processing CreateTablet for tablet abaf9caf88e44b7aa604b8429d98a36d (DEFAULT_TABLE table=test-table [id=a1147afc297a4223955cf8ccb8742be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:47.862805 27147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abaf9caf88e44b7aa604b8429d98a36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:47.865674 27236 tablet_bootstrap.cc:492] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534: Bootstrap starting.
I20260619 15:02:47.868603 27237 tablet_bootstrap.cc:492] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62: Bootstrap starting.
I20260619 15:02:47.869910 27236 tablet_bootstrap.cc:654] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:47.871290 27236 log.cc:824] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:47.872190 27237 tablet_bootstrap.cc:654] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:47.873747 27237 log.cc:824] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:47.874619 27236 tablet_bootstrap.cc:492] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534: No bootstrap required, opened a new log
I20260619 15:02:47.874945 27236 ts_tablet_manager.cc:1397] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 15:02:47.876914 27237 tablet_bootstrap.cc:492] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62: No bootstrap required, opened a new log
I20260619 15:02:47.877143 27240 tablet_bootstrap.cc:492] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be: Bootstrap starting.
I20260619 15:02:47.877316 27237 ts_tablet_manager.cc:1397] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 15:02:47.880386 27240 tablet_bootstrap.cc:654] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:47.881744 27240 log.cc:824] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:47.884567 27240 tablet_bootstrap.cc:492] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be: No bootstrap required, opened a new log
I20260619 15:02:47.884853 27240 ts_tablet_manager.cc:1397] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260619 15:02:47.885907 27236 raft_consensus.cc:348] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:47.886358 27236 raft_consensus.cc:374] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:47.886508 27236 raft_consensus.cc:729] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b4bb20a254493a141175daf679534, State: Initialized, Role: FOLLOWER
I20260619 15:02:47.887174 27236 consensus_queue.cc:260] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:47.888090 27237 raft_consensus.cc:348] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:47.888546 27237 raft_consensus.cc:374] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:47.888685 27237 raft_consensus.cc:729] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e781bf4699d49288db3eb6c3c47ee62, State: Initialized, Role: FOLLOWER
I20260619 15:02:47.888969 27236 ts_tablet_manager.cc:1428] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260619 15:02:47.889494 27237 consensus_queue.cc:260] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:47.891665 27078 heartbeater.cc:503] Master 127.24.103.126:35505 was elected leader, sending a full tablet report...
I20260619 15:02:47.892974 27237 ts_tablet_manager.cc:1428] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62: Time spent starting tablet: real 0.016s user 0.007s sys 0.007s
I20260619 15:02:47.894408 27240 raft_consensus.cc:348] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:47.894909 27240 raft_consensus.cc:374] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:47.895058 27240 raft_consensus.cc:729] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec1d74847e4425181613c9e261b21be, State: Initialized, Role: FOLLOWER
I20260619 15:02:47.895622 27240 consensus_queue.cc:260] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:47.896727 27217 heartbeater.cc:503] Master 127.24.103.126:35505 was elected leader, sending a full tablet report...
I20260619 15:02:47.897307 27240 ts_tablet_manager.cc:1428] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be: Time spent starting tablet: real 0.012s user 0.004s sys 0.009s
W20260619 15:02:47.966930 26940 tablet.cc:2369] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:02:47.991564 27079 tablet.cc:2369] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:02:47.999876 27242 raft_consensus.cc:482] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:02:48.000157 27242 raft_consensus.cc:504] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.001818 27242 leader_election.cc:288] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843), fec1d74847e4425181613c9e261b21be (127.24.103.67:35907)
W20260619 15:02:48.006042 27218 tablet.cc:2369] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:02:48.009634 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.002207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34843 (local address 127.24.103.65:60449)
0619 15:02:48.002958 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:48.002980 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:48.003120 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:48.003615 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:48.003624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:48.003657 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:48.004115 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.004128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.005475 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.005481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.007422 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.007434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.008799 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.008806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.008983 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.009018 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 15:02:48.009120 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 15:02:48.009270 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":22,"thread_start_us":384,"threads_started":1}
I20260619 15:02:48.009673 27232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.002222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:60449 (local address 127.24.103.66:34843)
0619 15:02:48.002422 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:48.002426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:48.003085 (+ 659us) server_negotiation.cc:407] Connection header received
0619 15:02:48.003345 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:48.003353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:48.003451 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:48.003575 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:48.004277 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.005333 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.007615 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.008603 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.008725 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.008778 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:48.009212 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 15:02:48.009405 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 15:02:48.010264 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.003278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:53815 (local address 127.24.103.67:35907)
0619 15:02:48.003516 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:48.003525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:48.004038 (+ 513us) server_negotiation.cc:407] Connection header received
0619 15:02:48.004186 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:48.004193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:48.004284 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:48.004377 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:48.004878 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.005965 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.008207 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.009169 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.009309 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.009377 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:48.009807 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 15:02:48.009983 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 15:02:48.010886 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.003121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:35907 (local address 127.24.103.65:53815)
0619 15:02:48.003871 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:48.003888 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:48.004039 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:48.004389 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:48.004393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:48.004416 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:48.004717 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.004725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.006108 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.006113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.008052 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.008066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.009421 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.009428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.009600 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.009638 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 15:02:48.010461 (+ 823us) client_negotiation.cc:240] Negotiation successful
0619 15:02:48.010572 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":24,"thread_start_us":418,"threads_started":1}
I20260619 15:02:48.010887 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abaf9caf88e44b7aa604b8429d98a36d" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e781bf4699d49288db3eb6c3c47ee62" is_pre_election: true
I20260619 15:02:48.011474 27032 raft_consensus.cc:2454] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 0.
I20260619 15:02:48.011724 27167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abaf9caf88e44b7aa604b8429d98a36d" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec1d74847e4425181613c9e261b21be" is_pre_election: true
I20260619 15:02:48.012205 26820 leader_election.cc:302] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b4bb20a254493a141175daf679534, 0e781bf4699d49288db3eb6c3c47ee62; no voters:
I20260619 15:02:48.012240 27167 raft_consensus.cc:2454] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 0.
I20260619 15:02:48.012639 27242 raft_consensus.cc:2781] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:02:48.012789 27242 raft_consensus.cc:482] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:02:48.012897 27242 raft_consensus.cc:3037] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:48.015280 27242 raft_consensus.cc:504] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.015878 27242 leader_election.cc:288] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 election: Requested vote from peers 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843), fec1d74847e4425181613c9e261b21be (127.24.103.67:35907)
I20260619 15:02:48.016376 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abaf9caf88e44b7aa604b8429d98a36d" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e781bf4699d49288db3eb6c3c47ee62"
I20260619 15:02:48.016449 27167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abaf9caf88e44b7aa604b8429d98a36d" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec1d74847e4425181613c9e261b21be"
I20260619 15:02:48.016639 27032 raft_consensus.cc:3037] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:48.016665 27167 raft_consensus.cc:3037] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:48.019081 27167 raft_consensus.cc:2454] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 1.
I20260619 15:02:48.019121 27032 raft_consensus.cc:2454] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 1.
I20260619 15:02:48.019510 26820 leader_election.cc:302] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b4bb20a254493a141175daf679534, fec1d74847e4425181613c9e261b21be; no voters:
I20260619 15:02:48.019819 27242 raft_consensus.cc:2781] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:48.020259 27242 raft_consensus.cc:686] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 1 LEADER]: Becoming Leader. State: Replica: 0b7b4bb20a254493a141175daf679534, State: Running, Role: LEADER
I20260619 15:02:48.020856 27242 consensus_queue.cc:237] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.025493 26732 catalog_manager.cc:5495] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b7b4bb20a254493a141175daf679534 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "0b7b4bb20a254493a141175daf679534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } health_report { overall_health: UNKNOWN } } }
Jun 19 15:02:48 dist-test-slave-zpbw krb5kdc[26687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881365, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:48.085429 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.069750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35505 (local address 127.0.0.1:44270)
0619 15:02:48.070424 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:48.070442 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:48.070588 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:48.070985 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:48.070990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:48.071921 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:48.072293 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.072306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.073324 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.073330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.074956 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.074968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.075766 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.075773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.075949 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.076729 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:48.076751 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:48.079576 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:48.082746 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:48.082759 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:48.082766 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:48.083208 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:48.083597 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:48.083603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:48.083607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:48.083767 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:48.084312 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:48.084319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:48.084478 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 15:02:48.084647 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 15:02:48.084951 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":361,"threads_started":1}
I20260619 15:02:48.085464 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.069724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.24.103.126:35505)
0619 15:02:48.069948 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:48.069955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:48.070539 (+ 584us) server_negotiation.cc:407] Connection header received
0619 15:02:48.070716 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:48.070720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:48.070841 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:48.070934 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:48.072455 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.073198 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.075120 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.075583 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.075680 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.079781 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:48.079786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:48.079790 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:48.079831 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:48.082597 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:48.083327 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:48.083334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:48.083337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:48.083472 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:48.083883 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:48.083889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:48.083897 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:48.084174 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:48.084321 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:48.084735 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 15:02:48.085100 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":72}
I20260619 15:02:48.090466 26732 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:44270:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 15:02:48.091593 26732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:02:48.098304 26869 tablet_service.cc:1467] Processing CreateTablet for tablet 84f1654f1a134f509a3b36aad0b07e6a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc64edf5100d4c098576fde71bcda561]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 15:02:48.098433 27008 tablet_service.cc:1467] Processing CreateTablet for tablet 84f1654f1a134f509a3b36aad0b07e6a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc64edf5100d4c098576fde71bcda561]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 15:02:48.098560 27147 tablet_service.cc:1467] Processing CreateTablet for tablet 84f1654f1a134f509a3b36aad0b07e6a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc64edf5100d4c098576fde71bcda561]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 15:02:48.099416 26869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f1654f1a134f509a3b36aad0b07e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:48.099431 27008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f1654f1a134f509a3b36aad0b07e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:48.099431 27147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f1654f1a134f509a3b36aad0b07e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:48.105932 27236 tablet_bootstrap.cc:492] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: Bootstrap starting.
I20260619 15:02:48.106887 27237 tablet_bootstrap.cc:492] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62: Bootstrap starting.
I20260619 15:02:48.107445 27240 tablet_bootstrap.cc:492] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be: Bootstrap starting.
I20260619 15:02:48.107868 27236 tablet_bootstrap.cc:654] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:48.108644 27237 tablet_bootstrap.cc:654] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:48.108965 27240 tablet_bootstrap.cc:654] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:48.111521 27236 tablet_bootstrap.cc:492] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: No bootstrap required, opened a new log
I20260619 15:02:48.111543 27237 tablet_bootstrap.cc:492] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62: No bootstrap required, opened a new log
I20260619 15:02:48.111667 27240 tablet_bootstrap.cc:492] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be: No bootstrap required, opened a new log
I20260619 15:02:48.111692 27237 ts_tablet_manager.cc:1397] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260619 15:02:48.111686 27236 ts_tablet_manager.cc:1397] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 15:02:48.111913 27240 ts_tablet_manager.cc:1397] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 15:02:48.112521 27237 raft_consensus.cc:348] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.112519 27236 raft_consensus.cc:348] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.112797 27237 raft_consensus.cc:374] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:48.112821 27236 raft_consensus.cc:374] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:48.112707 27240 raft_consensus.cc:348] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.112890 27237 raft_consensus.cc:729] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e781bf4699d49288db3eb6c3c47ee62, State: Initialized, Role: FOLLOWER
I20260619 15:02:48.112910 27236 raft_consensus.cc:729] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b4bb20a254493a141175daf679534, State: Initialized, Role: FOLLOWER
I20260619 15:02:48.112960 27240 raft_consensus.cc:374] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:48.113054 27240 raft_consensus.cc:729] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec1d74847e4425181613c9e261b21be, State: Initialized, Role: FOLLOWER
I20260619 15:02:48.113157 27236 consensus_queue.cc:260] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.113152 27237 consensus_queue.cc:260] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.113300 27240 consensus_queue.cc:260] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.113714 27237 ts_tablet_manager.cc:1428] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 15:02:48.113725 27236 ts_tablet_manager.cc:1428] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 15:02:48.113843 27240 ts_tablet_manager.cc:1428] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 15:02:48.114144 27244 tablet_replica.cc:439] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } } }
I20260619 15:02:48.114509 27244 tablet_replica.cc:442] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:48.114857 27243 tablet_replica.cc:439] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } } }
I20260619 15:02:48.115063 27242 tablet_replica.cc:439] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } } }
I20260619 15:02:48.115263 27243 tablet_replica.cc:442] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:48.115367 27242 tablet_replica.cc:442] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:48.328088 27242 raft_consensus.cc:482] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:02:48.328349 27242 raft_consensus.cc:504] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.328986 27242 leader_election.cc:288] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843), fec1d74847e4425181613c9e261b21be (127.24.103.67:35907)
I20260619 15:02:48.329615 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f1654f1a134f509a3b36aad0b07e6a" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e781bf4699d49288db3eb6c3c47ee62" is_pre_election: true
I20260619 15:02:48.329703 27167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f1654f1a134f509a3b36aad0b07e6a" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec1d74847e4425181613c9e261b21be" is_pre_election: true
I20260619 15:02:48.329914 27032 raft_consensus.cc:2454] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 0.
I20260619 15:02:48.329970 27167 raft_consensus.cc:2454] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 0.
I20260619 15:02:48.330396 26820 leader_election.cc:302] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b4bb20a254493a141175daf679534, fec1d74847e4425181613c9e261b21be; no voters:
I20260619 15:02:48.330711 27242 raft_consensus.cc:2781] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:02:48.330871 27242 raft_consensus.cc:482] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:02:48.331003 27242 raft_consensus.cc:3037] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:48.333003 27242 raft_consensus.cc:504] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.333549 27242 leader_election.cc:288] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 election: Requested vote from peers 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843), fec1d74847e4425181613c9e261b21be (127.24.103.67:35907)
I20260619 15:02:48.334136 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f1654f1a134f509a3b36aad0b07e6a" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e781bf4699d49288db3eb6c3c47ee62"
I20260619 15:02:48.334143 27167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f1654f1a134f509a3b36aad0b07e6a" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec1d74847e4425181613c9e261b21be"
I20260619 15:02:48.334365 27032 raft_consensus.cc:3037] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:48.334375 27167 raft_consensus.cc:3037] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:48.336580 27167 raft_consensus.cc:2454] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 1.
I20260619 15:02:48.336707 27032 raft_consensus.cc:2454] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 1.
I20260619 15:02:48.337030 26820 leader_election.cc:302] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b4bb20a254493a141175daf679534, fec1d74847e4425181613c9e261b21be; no voters:
I20260619 15:02:48.337419 27242 raft_consensus.cc:2781] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:48.337651 27242 raft_consensus.cc:686] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [term 1 LEADER]: Becoming Leader. State: Replica: 0b7b4bb20a254493a141175daf679534, State: Running, Role: LEADER
I20260619 15:02:48.337898 27242 consensus_queue.cc:237] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.339486 27247 tablet_replica.cc:439] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: TxnStatusTablet state changed. Reason: New leader 0b7b4bb20a254493a141175daf679534. Latest consensus state: current_term: 1 leader_uuid: "0b7b4bb20a254493a141175daf679534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } } }
I20260619 15:02:48.339905 27247 tablet_replica.cc:442] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: This TxnStatusTablet replica's current role is: LEADER
I20260619 15:02:48.340672 27259 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 15:02:48.340991 27259 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 15:02:48.341243 26732 catalog_manager.cc:5495] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b7b4bb20a254493a141175daf679534 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "0b7b4bb20a254493a141175daf679534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } health_report { overall_health: UNKNOWN } } }
I20260619 15:02:48.409178 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.402172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:32797 (local address 127.0.0.1:59852)
0619 15:02:48.402414 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:48.402441 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:48.402552 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:48.403437 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:48.403442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:48.403454 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 15:02:48.403658 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.403669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.405245 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.405248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.406394 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.406405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.407432 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.407436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.407583 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.407641 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 15:02:48.408753 (+ 1112us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 15:02:48.408768 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 15:02:48.408822 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 15:02:48.408941 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 15:02:48.409374 27262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.402382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.24.103.65:32797)
0619 15:02:48.403047 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:48.403054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:48.403074 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:02:48.403179 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:48.403182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:48.403256 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:48.403422 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:02:48.403835 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.405106 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.406538 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.407252 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.407405 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.407837 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:02:48.408644 (+ 807us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:02:48.408815 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:48.408866 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 15:02:48.408954 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":387,"threads_started":1}
I20260619 15:02:48.417598 27032 raft_consensus.cc:1261] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Refusing update from remote peer 0b7b4bb20a254493a141175daf679534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:48.417598 27167 raft_consensus.cc:1261] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Refusing update from remote peer 0b7b4bb20a254493a141175daf679534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:48.418670 27247 consensus_queue.cc:1035] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:48.419423 27242 consensus_queue.cc:1035] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:48.433542 27243 tablet_replica.cc:439] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62: TxnStatusTablet state changed. Reason: New leader 0b7b4bb20a254493a141175daf679534. Latest consensus state: current_term: 1 leader_uuid: "0b7b4bb20a254493a141175daf679534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } } }
I20260619 15:02:48.433920 27243 tablet_replica.cc:442] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:48.434309 27244 tablet_replica.cc:439] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be: TxnStatusTablet state changed. Reason: New leader 0b7b4bb20a254493a141175daf679534. Latest consensus state: current_term: 1 leader_uuid: "0b7b4bb20a254493a141175daf679534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } } }
I20260619 15:02:48.434568 27244 tablet_replica.cc:442] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:48.439184 27247 tablet_replica.cc:439] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b7b4bb20a254493a141175daf679534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } } }
I20260619 15:02:48.439459 27247 tablet_replica.cc:442] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: This TxnStatusTablet replica's current role is: LEADER
I20260619 15:02:48.441584 27242 tablet_replica.cc:439] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b7b4bb20a254493a141175daf679534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } } }
I20260619 15:02:48.443300 27264 mvcc.cc:205] Tried to move back new op lower bound from 7298586085027356672 to 7298586084713889792. Current Snapshot: MvccSnapshot[applied={T|T < 7298586085027356672}]
I20260619 15:02:48.443559 27242 tablet_replica.cc:442] T 84f1654f1a134f509a3b36aad0b07e6a P 0b7b4bb20a254493a141175daf679534: This TxnStatusTablet replica's current role is: LEADER
I20260619 15:02:48.447453 27244 tablet_replica.cc:439] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b7b4bb20a254493a141175daf679534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } } }
I20260619 15:02:48.447872 27244 tablet_replica.cc:442] T 84f1654f1a134f509a3b36aad0b07e6a P fec1d74847e4425181613c9e261b21be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:48.450582 27243 tablet_replica.cc:439] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b7b4bb20a254493a141175daf679534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } } }
I20260619 15:02:48.450862 27243 tablet_replica.cc:442] T 84f1654f1a134f509a3b36aad0b07e6a P 0e781bf4699d49288db3eb6c3c47ee62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:02:48.463959 27262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.457999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.24.103.65:32797)
0619 15:02:48.458224 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:48.458230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:48.458615 (+ 385us) server_negotiation.cc:407] Connection header received
0619 15:02:48.458796 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:48.458803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:48.458888 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:48.458991 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:02:48.459360 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.460508 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.461849 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.462572 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.462720 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.463162 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:02:48.463298 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:02:48.463387 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:48.463540 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 15:02:48.463627 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:02:48.481247 27262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.475635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:57105 (local address 127.24.103.65:32797)
0619 15:02:48.475841 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:48.475849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:48.476192 (+ 343us) server_negotiation.cc:407] Connection header received
0619 15:02:48.476413 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:48.476416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:48.476493 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:48.476583 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:02:48.477075 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.477854 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.479388 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.479814 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.479960 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.480386 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:02:48.480482 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:02:48.480582 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:48.480726 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 15:02:48.480846 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":60}
I20260619 15:02:48.481361 27275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.475558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:32797 (local address 127.24.103.65:57105)
0619 15:02:48.476097 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:48.476112 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:48.476276 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:48.476592 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:48.476596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:48.476605 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 15:02:48.476946 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.476954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.477991 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.477997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.479254 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.479266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.480002 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.480010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.480133 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.480182 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 15:02:48.480579 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 15:02:48.480600 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 15:02:48.480708 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 15:02:48.480888 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":18,"thread_start_us":284,"threads_started":1}
I20260619 15:02:48.492403 27167 raft_consensus.cc:1261] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Refusing update from remote peer 0b7b4bb20a254493a141175daf679534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:48.492542 27032 raft_consensus.cc:1261] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Refusing update from remote peer 0b7b4bb20a254493a141175daf679534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:48.492935 27247 consensus_queue.cc:1035] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:48.493220 27242 consensus_queue.cc:1035] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:48.497754 27263 mvcc.cc:205] Tried to move back new op lower bound from 7298586085340049408 to 7298586083417686016. Current Snapshot: MvccSnapshot[applied={T|T < 7298586085340049408}]
I20260619 15:02:48.556937 26732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:02:48.557202 26732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:02:48.561013 26732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e: Sending DeleteTablet for 3 replicas of tablet abaf9caf88e44b7aa604b8429d98a36d
I20260619 15:02:48.562307 27008 tablet_service.cc:1514] Processing DeleteTablet for tablet abaf9caf88e44b7aa604b8429d98a36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:48 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:42588
I20260619 15:02:48.562630 27147 tablet_service.cc:1514] Processing DeleteTablet for tablet abaf9caf88e44b7aa604b8429d98a36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:48 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:55002
I20260619 15:02:48.563195 26868 tablet_service.cc:1514] Processing DeleteTablet for tablet abaf9caf88e44b7aa604b8429d98a36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:48 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:59840
I20260619 15:02:48.564209 26732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:02:48.565340 26732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:02:48.567368 27281 tablet_replica.cc:330] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be: stopping tablet replica
I20260619 15:02:48.567474 27280 tablet_replica.cc:330] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62: stopping tablet replica
I20260619 15:02:48.568048 27282 tablet_replica.cc:330] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534: stopping tablet replica
I20260619 15:02:48.567996 27280 raft_consensus.cc:2229] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:02:48.568383 27280 raft_consensus.cc:2258] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:48.568862 27281 raft_consensus.cc:2229] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:02:48.569108 27282 raft_consensus.cc:2229] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:02:48.569710 27282 raft_consensus.cc:2258] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:48.570246 27281 raft_consensus.cc:2258] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:48.571766 27282 ts_tablet_manager.cc:1905] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:48.572965 27280 ts_tablet_manager.cc:1905] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:48.574766 27008 tablet_service.cc:1467] Processing CreateTablet for tablet 2772a105ac9e4dd3befe91a835e323d4 (DEFAULT_TABLE table=test-table [id=71f50b30a7384cdb9083ea4716acc5b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:48.575075 27147 tablet_service.cc:1467] Processing CreateTablet for tablet 2772a105ac9e4dd3befe91a835e323d4 (DEFAULT_TABLE table=test-table [id=71f50b30a7384cdb9083ea4716acc5b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:48.575282 27008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2772a105ac9e4dd3befe91a835e323d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:48.575538 27147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2772a105ac9e4dd3befe91a835e323d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:48.579784 26868 tablet_service.cc:1467] Processing CreateTablet for tablet 2772a105ac9e4dd3befe91a835e323d4 (DEFAULT_TABLE table=test-table [id=71f50b30a7384cdb9083ea4716acc5b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:48.580350 26868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2772a105ac9e4dd3befe91a835e323d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:48.580793 27282 ts_tablet_manager.cc:1918] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:02:48.581032 27282 log.cc:1196] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/abaf9caf88e44b7aa604b8429d98a36d
I20260619 15:02:48.581683 27282 ts_tablet_manager.cc:1939] T abaf9caf88e44b7aa604b8429d98a36d P 0b7b4bb20a254493a141175daf679534: Deleting consensus metadata
I20260619 15:02:48.582479 27237 tablet_bootstrap.cc:492] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62: Bootstrap starting.
I20260619 15:02:48.583815 26718 catalog_manager.cc:4841] TS 0b7b4bb20a254493a141175daf679534 (127.24.103.65:32797): tablet abaf9caf88e44b7aa604b8429d98a36d (table test-table [id=a1147afc297a4223955cf8ccb8742be2]) successfully deleted
I20260619 15:02:48.584249 27237 tablet_bootstrap.cc:654] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:48.586462 27237 tablet_bootstrap.cc:492] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62: No bootstrap required, opened a new log
I20260619 15:02:48.586478 27281 ts_tablet_manager.cc:1905] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:48.586826 27237 ts_tablet_manager.cc:1397] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 15:02:48.587658 27237 raft_consensus.cc:348] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.588002 27237 raft_consensus.cc:374] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:48.588120 27237 raft_consensus.cc:729] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e781bf4699d49288db3eb6c3c47ee62, State: Initialized, Role: FOLLOWER
I20260619 15:02:48.588423 27237 consensus_queue.cc:260] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.589736 27237 ts_tablet_manager.cc:1428] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 15:02:48.591122 27240 tablet_bootstrap.cc:492] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be: Bootstrap starting.
I20260619 15:02:48.593027 27240 tablet_bootstrap.cc:654] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:48.596554 27281 ts_tablet_manager.cc:1918] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:02:48.596736 27281 log.cc:1196] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/abaf9caf88e44b7aa604b8429d98a36d
I20260619 15:02:48.597168 27281 ts_tablet_manager.cc:1939] T abaf9caf88e44b7aa604b8429d98a36d P fec1d74847e4425181613c9e261b21be: Deleting consensus metadata
I20260619 15:02:48.597859 27236 tablet_bootstrap.cc:492] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534: Bootstrap starting.
I20260619 15:02:48.599006 26717 catalog_manager.cc:4841] TS fec1d74847e4425181613c9e261b21be (127.24.103.67:35907): tablet abaf9caf88e44b7aa604b8429d98a36d (table test-table [id=a1147afc297a4223955cf8ccb8742be2]) successfully deleted
I20260619 15:02:48.599625 27236 tablet_bootstrap.cc:654] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:48.602113 27236 tablet_bootstrap.cc:492] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534: No bootstrap required, opened a new log
I20260619 15:02:48.602334 27236 ts_tablet_manager.cc:1397] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 15:02:48.603140 27236 raft_consensus.cc:348] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.603442 27236 raft_consensus.cc:374] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:48.603539 27236 raft_consensus.cc:729] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b4bb20a254493a141175daf679534, State: Initialized, Role: FOLLOWER
I20260619 15:02:48.603814 27236 consensus_queue.cc:260] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.604579 27236 ts_tablet_manager.cc:1428] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 15:02:48.605511 27280 ts_tablet_manager.cc:1918] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:02:48.605693 27280 log.cc:1196] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/abaf9caf88e44b7aa604b8429d98a36d
I20260619 15:02:48.606261 27280 ts_tablet_manager.cc:1939] T abaf9caf88e44b7aa604b8429d98a36d P 0e781bf4699d49288db3eb6c3c47ee62: Deleting consensus metadata
I20260619 15:02:48.607978 27240 tablet_bootstrap.cc:492] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be: No bootstrap required, opened a new log
I20260619 15:02:48.608173 27240 ts_tablet_manager.cc:1397] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
I20260619 15:02:48.608788 26717 catalog_manager.cc:4841] TS 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843): tablet abaf9caf88e44b7aa604b8429d98a36d (table test-table [id=a1147afc297a4223955cf8ccb8742be2]) successfully deleted
I20260619 15:02:48.608995 27240 raft_consensus.cc:348] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.609254 27240 raft_consensus.cc:374] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:48.609339 27240 raft_consensus.cc:729] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec1d74847e4425181613c9e261b21be, State: Initialized, Role: FOLLOWER
I20260619 15:02:48.609598 27240 consensus_queue.cc:260] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.610102 27240 ts_tablet_manager.cc:1428] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 15:02:48.673208 27244 raft_consensus.cc:482] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:02:48.673475 27244 raft_consensus.cc:504] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.675302 27244 leader_election.cc:288] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b7b4bb20a254493a141175daf679534 (127.24.103.65:32797), 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843)
I20260619 15:02:48.687547 27262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.675701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:34485 (local address 127.24.103.65:32797)
0619 15:02:48.675948 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:48.675958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:48.676552 (+ 594us) server_negotiation.cc:407] Connection header received
0619 15:02:48.676884 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:48.676892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:48.677001 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:48.677077 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:48.677949 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.679529 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.683231 (+ 3702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.685224 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.685420 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.685537 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:48.686904 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0619 15:02:48.687121 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 15:02:48.688498 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.675472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:32797 (local address 127.24.103.67:34485)
0619 15:02:48.676417 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:48.676445 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:48.676691 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:48.677335 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:48.677342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:48.677385 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:48.677724 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.677739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.679799 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.679807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.682901 (+ 3094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.682925 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.685720 (+ 2795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.685731 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.686031 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.686086 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 15:02:48.687921 (+ 1835us) client_negotiation.cc:240] Negotiation successful
0619 15:02:48.688139 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":12610,"threads_started":1}
I20260619 15:02:48.689612 26889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2772a105ac9e4dd3befe91a835e323d4" candidate_uuid: "fec1d74847e4425181613c9e261b21be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7b4bb20a254493a141175daf679534" is_pre_election: true
I20260619 15:02:48.690218 26889 raft_consensus.cc:2454] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec1d74847e4425181613c9e261b21be in term 0.
I20260619 15:02:48.691052 27100 leader_election.cc:302] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b4bb20a254493a141175daf679534, fec1d74847e4425181613c9e261b21be; no voters:
I20260619 15:02:48.691599 27244 raft_consensus.cc:2781] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:02:48.691757 27244 raft_consensus.cc:482] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:02:48.691862 27244 raft_consensus.cc:3037] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:48.695204 27244 raft_consensus.cc:504] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.695897 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.676126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:60007 (local address 127.24.103.66:34843)
0619 15:02:48.686724 (+ 10598us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:48.686733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:48.688599 (+ 1866us) server_negotiation.cc:407] Connection header received
0619 15:02:48.689054 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:48.689059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:48.689149 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:48.689259 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:02:48.689758 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.690600 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.693352 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.694635 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.694827 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.694909 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:48.695367 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 15:02:48.695515 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10398,"thread_start_us":10269,"threads_started":1}
I20260619 15:02:48.696048 27244 leader_election.cc:288] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [CANDIDATE]: Term 1 election: Requested vote from peers 0b7b4bb20a254493a141175daf679534 (127.24.103.65:32797), 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843)
I20260619 15:02:48.696178 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.675678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34843 (local address 127.24.103.67:60007)
0619 15:02:48.688440 (+ 12762us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:48.688465 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:48.688871 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:48.689287 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:48.689292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:48.689319 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:02:48.689588 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.689595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.690891 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.690895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.693147 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.693161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.694915 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.694923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.695136 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.695178 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 15:02:48.695317 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 15:02:48.695563 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12606,"thread_start_us":671,"threads_started":1}
I20260619 15:02:48.696717 26889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2772a105ac9e4dd3befe91a835e323d4" candidate_uuid: "fec1d74847e4425181613c9e261b21be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7b4bb20a254493a141175daf679534"
I20260619 15:02:48.696944 26889 raft_consensus.cc:3037] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:48.697086 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2772a105ac9e4dd3befe91a835e323d4" candidate_uuid: "fec1d74847e4425181613c9e261b21be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e781bf4699d49288db3eb6c3c47ee62" is_pre_election: true
I20260619 15:02:48.697115 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2772a105ac9e4dd3befe91a835e323d4" candidate_uuid: "fec1d74847e4425181613c9e261b21be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e781bf4699d49288db3eb6c3c47ee62"
I20260619 15:02:48.697444 27031 raft_consensus.cc:2454] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec1d74847e4425181613c9e261b21be in term 0.
I20260619 15:02:48.699759 26889 raft_consensus.cc:2454] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec1d74847e4425181613c9e261b21be in term 1.
I20260619 15:02:48.700213 27100 leader_election.cc:302] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b7b4bb20a254493a141175daf679534, fec1d74847e4425181613c9e261b21be; no voters: 0e781bf4699d49288db3eb6c3c47ee62
I20260619 15:02:48.700570 27244 raft_consensus.cc:2781] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:48.701083 27244 raft_consensus.cc:686] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 1 LEADER]: Becoming Leader. State: Replica: fec1d74847e4425181613c9e261b21be, State: Running, Role: LEADER
I20260619 15:02:48.701464 27244 consensus_queue.cc:237] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:48.705364 26732 catalog_manager.cc:5495] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be reported cstate change: term changed from 0 to 1, leader changed from <none> to fec1d74847e4425181613c9e261b21be (127.24.103.67). New cstate: current_term: 1 leader_uuid: "fec1d74847e4425181613c9e261b21be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } health_report { overall_health: HEALTHY } } }
I20260619 15:02:48.744236 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.736315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.24.103.67:35907)
0619 15:02:48.737010 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:48.737016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:48.737038 (+ 22us) server_negotiation.cc:407] Connection header received
0619 15:02:48.737142 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:48.737147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:48.737264 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:48.737474 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:02:48.738047 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.739302 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.741441 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.742091 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.742222 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.742844 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:02:48.743512 (+ 668us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:02:48.743617 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:48.743812 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 15:02:48.743901 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":424,"threads_started":1}
I20260619 15:02:48.761976 27262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.755862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:34123 (local address 127.24.103.65:32797)
0619 15:02:48.756038 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:48.756044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:48.756743 (+ 699us) server_negotiation.cc:407] Connection header received
0619 15:02:48.756963 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:48.756969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:48.757066 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:48.757167 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:02:48.757664 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.758472 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.760108 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.760608 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.760758 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.761166 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:02:48.761288 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:02:48.761385 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:48.761519 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 15:02:48.761611 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 15:02:48.762085 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.755989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:32797 (local address 127.24.103.67:34123)
0619 15:02:48.756621 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:48.756641 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:48.756819 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:48.757181 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:48.757186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:48.757197 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 15:02:48.757518 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.757527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.758601 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.758608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.759953 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.759964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.760797 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.760805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.760930 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.760982 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 15:02:48.761385 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 15:02:48.761398 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 15:02:48.761499 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 15:02:48.761633 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":312,"threads_started":1}
I20260619 15:02:48.770800 27031 raft_consensus.cc:3037] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:48.770958 26889 raft_consensus.cc:1261] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Refusing update from remote peer fec1d74847e4425181613c9e261b21be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:48.771739 27289 consensus_queue.cc:1035] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:48.773833 27031 raft_consensus.cc:1261] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Refusing update from remote peer fec1d74847e4425181613c9e261b21be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:48.774394 27289 consensus_queue.cc:1035] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:48.781134 27265 mvcc.cc:205] Tried to move back new op lower bound from 7298586086477897728 to 7298586086204825600. Current Snapshot: MvccSnapshot[applied={T|T < 7298586086477897728}]
I20260619 15:02:48.809875 27275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.801755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:35907 (local address 127.24.103.65:45291)
0619 15:02:48.802013 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:48.802028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:48.802164 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:48.802645 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:48.802649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:48.802662 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 15:02:48.802895 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.802910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.804459 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.804467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.806440 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:48.806458 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.808416 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.808424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:48.808662 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.808728 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 15:02:48.809259 (+ 531us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 15:02:48.809272 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 15:02:48.809352 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 15:02:48.809512 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260619 15:02:48.810616 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:48.801620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:45291 (local address 127.24.103.67:35907)
0619 15:02:48.801806 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:48.801814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:48.802101 (+ 287us) server_negotiation.cc:407] Connection header received
0619 15:02:48.802301 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:48.802306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:48.802384 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:48.802462 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:02:48.803054 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.804248 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.807515 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:48.808149 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:48.808284 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:48.808936 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:02:48.809081 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:02:48.809166 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:48.810185 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0619 15:02:48.810272 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 15:02:48.811808 27264 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 15:02:50.399268 26732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:02:50.399487 26732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:02:50.404156 26732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e: Sending DeleteTablet for 3 replicas of tablet 2772a105ac9e4dd3befe91a835e323d4
I20260619 15:02:50.405072 27147 tablet_service.cc:1514] Processing DeleteTablet for tablet 2772a105ac9e4dd3befe91a835e323d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:50 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:55002
I20260619 15:02:50.405048 26868 tablet_service.cc:1514] Processing DeleteTablet for tablet 2772a105ac9e4dd3befe91a835e323d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:50 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:59840
I20260619 15:02:50.405215 27008 tablet_service.cc:1514] Processing DeleteTablet for tablet 2772a105ac9e4dd3befe91a835e323d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:50 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:42588
I20260619 15:02:50.417981 27322 tablet_replica.cc:330] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534: stopping tablet replica
I20260619 15:02:50.418260 27322 raft_consensus.cc:2229] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:02:50.418536 27322 raft_consensus.cc:2258] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:50.419076 27324 tablet_replica.cc:330] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62: stopping tablet replica
I20260619 15:02:50.419310 27323 tablet_replica.cc:330] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be: stopping tablet replica
I20260619 15:02:50.419492 27324 raft_consensus.cc:2229] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:02:50.419607 27323 raft_consensus.cc:2229] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:02:50.419831 27324 raft_consensus.cc:2258] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:50.420323 27323 raft_consensus.cc:2258] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:50.420460 27322 ts_tablet_manager.cc:1905] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:50.420935 27324 ts_tablet_manager.cc:1905] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:50.421288 27323 ts_tablet_manager.cc:1905] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:50.425506 27324 ts_tablet_manager.cc:1918] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 15:02:50.425654 27324 log.cc:1196] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/2772a105ac9e4dd3befe91a835e323d4
I20260619 15:02:50.426153 27324 ts_tablet_manager.cc:1939] T 2772a105ac9e4dd3befe91a835e323d4 P 0e781bf4699d49288db3eb6c3c47ee62: Deleting consensus metadata
I20260619 15:02:50.427587 26717 catalog_manager.cc:4841] TS 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843): tablet 2772a105ac9e4dd3befe91a835e323d4 (table test-table [id=71f50b30a7384cdb9083ea4716acc5b0]) successfully deleted
I20260619 15:02:50.427788 27322 ts_tablet_manager.cc:1918] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 15:02:50.427969 27322 log.cc:1196] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/2772a105ac9e4dd3befe91a835e323d4
I20260619 15:02:50.428658 27323 ts_tablet_manager.cc:1918] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 15:02:50.428802 27323 log.cc:1196] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/2772a105ac9e4dd3befe91a835e323d4
I20260619 15:02:50.428921 27322 ts_tablet_manager.cc:1939] T 2772a105ac9e4dd3befe91a835e323d4 P 0b7b4bb20a254493a141175daf679534: Deleting consensus metadata
I20260619 15:02:50.429260 27323 ts_tablet_manager.cc:1939] T 2772a105ac9e4dd3befe91a835e323d4 P fec1d74847e4425181613c9e261b21be: Deleting consensus metadata
I20260619 15:02:50.430151 26718 catalog_manager.cc:4841] TS 0b7b4bb20a254493a141175daf679534 (127.24.103.65:32797): tablet 2772a105ac9e4dd3befe91a835e323d4 (table test-table [id=71f50b30a7384cdb9083ea4716acc5b0]) successfully deleted
I20260619 15:02:50.430317 26717 catalog_manager.cc:4841] TS fec1d74847e4425181613c9e261b21be (127.24.103.67:35907): tablet 2772a105ac9e4dd3befe91a835e323d4 (table test-table [id=71f50b30a7384cdb9083ea4716acc5b0]) successfully deleted
Jun 19 15:02:50 dist-test-slave-zpbw krb5kdc[26687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 15:02:50 dist-test-slave-zpbw krb5kdc[26687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:50.467324 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:50.453400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.24.103.126:35505)
0619 15:02:50.454020 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:50.454025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:50.454058 (+ 33us) server_negotiation.cc:407] Connection header received
0619 15:02:50.454153 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:50.454157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:50.454222 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:50.454326 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:50.455721 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:50.456547 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:50.457759 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:50.458187 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:50.458301 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:50.462123 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:50.462128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:50.462132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:50.462167 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:50.464932 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:50.465611 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:50.465615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:50.465617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:50.465742 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:50.466094 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:50.466098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:50.466100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:50.466378 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:50.466559 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:50.466858 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 15:02:50.467059 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":339,"threads_started":1}
I20260619 15:02:50.471112 26732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:02:50.471865 26732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:02:50.478359 26868 tablet_service.cc:1467] Processing CreateTablet for tablet d950b2ee34c8424e9a2a633941c410ca (DEFAULT_TABLE table=test-table [id=18251fed664d4f6c9bb4481b7b3d6400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:50.478365 27008 tablet_service.cc:1467] Processing CreateTablet for tablet d950b2ee34c8424e9a2a633941c410ca (DEFAULT_TABLE table=test-table [id=18251fed664d4f6c9bb4481b7b3d6400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:50.478677 27147 tablet_service.cc:1467] Processing CreateTablet for tablet d950b2ee34c8424e9a2a633941c410ca (DEFAULT_TABLE table=test-table [id=18251fed664d4f6c9bb4481b7b3d6400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:50.478852 26868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d950b2ee34c8424e9a2a633941c410ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:50.478865 27008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d950b2ee34c8424e9a2a633941c410ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:50.479121 27147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d950b2ee34c8424e9a2a633941c410ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:50.486138 27335 tablet_bootstrap.cc:492] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534: Bootstrap starting.
I20260619 15:02:50.486224 27334 tablet_bootstrap.cc:492] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be: Bootstrap starting.
I20260619 15:02:50.487308 27333 tablet_bootstrap.cc:492] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62: Bootstrap starting.
I20260619 15:02:50.487912 27335 tablet_bootstrap.cc:654] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:50.488754 27334 tablet_bootstrap.cc:654] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:50.489141 27333 tablet_bootstrap.cc:654] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:50.490427 27335 tablet_bootstrap.cc:492] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534: No bootstrap required, opened a new log
I20260619 15:02:50.490648 27335 ts_tablet_manager.cc:1397] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 15:02:50.490900 27334 tablet_bootstrap.cc:492] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be: No bootstrap required, opened a new log
I20260619 15:02:50.491071 27334 ts_tablet_manager.cc:1397] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 15:02:50.491417 27335 raft_consensus.cc:348] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.491701 27335 raft_consensus.cc:374] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:50.491796 27335 raft_consensus.cc:729] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b4bb20a254493a141175daf679534, State: Initialized, Role: FOLLOWER
I20260619 15:02:50.491869 27333 tablet_bootstrap.cc:492] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62: No bootstrap required, opened a new log
I20260619 15:02:50.492019 27333 ts_tablet_manager.cc:1397] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 15:02:50.492041 27335 consensus_queue.cc:260] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.492517 27334 raft_consensus.cc:348] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.492841 27334 raft_consensus.cc:374] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:50.492944 27334 raft_consensus.cc:729] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec1d74847e4425181613c9e261b21be, State: Initialized, Role: FOLLOWER
I20260619 15:02:50.492830 27333 raft_consensus.cc:348] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.493124 27333 raft_consensus.cc:374] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:50.493242 27333 raft_consensus.cc:729] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e781bf4699d49288db3eb6c3c47ee62, State: Initialized, Role: FOLLOWER
I20260619 15:02:50.493324 27335 ts_tablet_manager.cc:1428] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 15:02:50.493355 27334 consensus_queue.cc:260] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.493479 27333 consensus_queue.cc:260] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.494092 27334 ts_tablet_manager.cc:1428] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 15:02:50.494825 27333 ts_tablet_manager.cc:1428] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 15:02:50.513340 27299 raft_consensus.cc:482] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:02:50.513542 27299 raft_consensus.cc:504] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.514189 27299 leader_election.cc:288] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843), fec1d74847e4425181613c9e261b21be (127.24.103.67:35907)
I20260619 15:02:50.514810 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d950b2ee34c8424e9a2a633941c410ca" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e781bf4699d49288db3eb6c3c47ee62" is_pre_election: true
I20260619 15:02:50.515106 27032 raft_consensus.cc:2454] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 0.
I20260619 15:02:50.515349 27166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d950b2ee34c8424e9a2a633941c410ca" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec1d74847e4425181613c9e261b21be" is_pre_election: true
I20260619 15:02:50.515611 27166 raft_consensus.cc:2454] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 0.
I20260619 15:02:50.515622 26820 leader_election.cc:302] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b4bb20a254493a141175daf679534, 0e781bf4699d49288db3eb6c3c47ee62; no voters:
I20260619 15:02:50.515931 27299 raft_consensus.cc:2781] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:02:50.516072 27299 raft_consensus.cc:482] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:02:50.516184 27299 raft_consensus.cc:3037] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:50.518270 27299 raft_consensus.cc:504] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.518909 27299 leader_election.cc:288] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 election: Requested vote from peers 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843), fec1d74847e4425181613c9e261b21be (127.24.103.67:35907)
I20260619 15:02:50.519336 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d950b2ee34c8424e9a2a633941c410ca" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e781bf4699d49288db3eb6c3c47ee62"
I20260619 15:02:50.519384 27166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d950b2ee34c8424e9a2a633941c410ca" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec1d74847e4425181613c9e261b21be"
I20260619 15:02:50.519554 27032 raft_consensus.cc:3037] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:50.519593 27166 raft_consensus.cc:3037] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:50.521852 27032 raft_consensus.cc:2454] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 1.
I20260619 15:02:50.521903 27166 raft_consensus.cc:2454] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 1.
I20260619 15:02:50.522258 26820 leader_election.cc:302] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b4bb20a254493a141175daf679534, fec1d74847e4425181613c9e261b21be; no voters:
I20260619 15:02:50.522535 27299 raft_consensus.cc:2781] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:50.522706 27299 raft_consensus.cc:686] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 1 LEADER]: Becoming Leader. State: Replica: 0b7b4bb20a254493a141175daf679534, State: Running, Role: LEADER
I20260619 15:02:50.522995 27299 consensus_queue.cc:237] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.525552 26732 catalog_manager.cc:5495] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b7b4bb20a254493a141175daf679534 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "0b7b4bb20a254493a141175daf679534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } health_report { overall_health: UNKNOWN } } }
I20260619 15:02:50.549003 27342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:50.543519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.24.103.65:32797)
0619 15:02:50.544107 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:50.544111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:50.544130 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:02:50.544183 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:50.544187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:50.544249 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:50.544330 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:02:50.544693 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:50.545447 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:50.546845 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:50.547783 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:50.547917 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:50.548279 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:02:50.548388 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:02:50.548457 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:50.548576 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 15:02:50.548639 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":310,"threads_started":1}
I20260619 15:02:50.557940 27166 raft_consensus.cc:1261] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Refusing update from remote peer 0b7b4bb20a254493a141175daf679534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:50.558243 27032 raft_consensus.cc:1261] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Refusing update from remote peer 0b7b4bb20a254493a141175daf679534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:50.558475 27307 consensus_queue.cc:1035] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:50.558794 27299 consensus_queue.cc:1035] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:50.601029 26732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:02:50.601241 26732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:02:50.604153 26732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e: Sending DeleteTablet for 3 replicas of tablet d950b2ee34c8424e9a2a633941c410ca
I20260619 15:02:50.605015 26869 tablet_service.cc:1514] Processing DeleteTablet for tablet d950b2ee34c8424e9a2a633941c410ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:50 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:59840
I20260619 15:02:50.605223 27147 tablet_service.cc:1514] Processing DeleteTablet for tablet d950b2ee34c8424e9a2a633941c410ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:50 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:55002
I20260619 15:02:50.605253 27008 tablet_service.cc:1514] Processing DeleteTablet for tablet d950b2ee34c8424e9a2a633941c410ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:50 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:42588
I20260619 15:02:50.605355 27322 tablet_replica.cc:330] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534: stopping tablet replica
I20260619 15:02:50.605558 27324 tablet_replica.cc:330] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62: stopping tablet replica
I20260619 15:02:50.605579 27322 raft_consensus.cc:2229] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:02:50.605760 27324 raft_consensus.cc:2229] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:02:50.605960 27322 raft_consensus.cc:2258] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:50.605990 27324 raft_consensus.cc:2258] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:50.606269 27323 tablet_replica.cc:330] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be: stopping tablet replica
I20260619 15:02:50.606473 27323 raft_consensus.cc:2229] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:02:50.606361 26732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 15:02:50.606666 27323 raft_consensus.cc:2258] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 15:02:50.607156 26732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:02:50.607247 27324 ts_tablet_manager.cc:1905] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:50.607769 27323 ts_tablet_manager.cc:1905] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:50.607899 27322 ts_tablet_manager.cc:1905] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:50.614174 27323 ts_tablet_manager.cc:1918] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:02:50.614391 27323 log.cc:1196] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/d950b2ee34c8424e9a2a633941c410ca
I20260619 15:02:50.614849 27323 ts_tablet_manager.cc:1939] T d950b2ee34c8424e9a2a633941c410ca P fec1d74847e4425181613c9e261b21be: Deleting consensus metadata
I20260619 15:02:50.615021 26869 tablet_service.cc:1467] Processing CreateTablet for tablet 843dac42be5a49d2b0601b35202c9b21 (DEFAULT_TABLE table=test-table [id=ab043df153a34ca2a19b0e028a81af71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:50.615355 27147 tablet_service.cc:1467] Processing CreateTablet for tablet 843dac42be5a49d2b0601b35202c9b21 (DEFAULT_TABLE table=test-table [id=ab043df153a34ca2a19b0e028a81af71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:50.615484 26869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843dac42be5a49d2b0601b35202c9b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:50.615548 27008 tablet_service.cc:1467] Processing CreateTablet for tablet 843dac42be5a49d2b0601b35202c9b21 (DEFAULT_TABLE table=test-table [id=ab043df153a34ca2a19b0e028a81af71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:02:50.615762 27147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843dac42be5a49d2b0601b35202c9b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:50.615937 27008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843dac42be5a49d2b0601b35202c9b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:50.618726 27324 ts_tablet_manager.cc:1918] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:02:50.618942 27324 log.cc:1196] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/d950b2ee34c8424e9a2a633941c410ca
I20260619 15:02:50.619226 26717 catalog_manager.cc:4841] TS fec1d74847e4425181613c9e261b21be (127.24.103.67:35907): tablet d950b2ee34c8424e9a2a633941c410ca (table test-table [id=18251fed664d4f6c9bb4481b7b3d6400]) successfully deleted
I20260619 15:02:50.619421 27324 ts_tablet_manager.cc:1939] T d950b2ee34c8424e9a2a633941c410ca P 0e781bf4699d49288db3eb6c3c47ee62: Deleting consensus metadata
I20260619 15:02:50.620698 26717 catalog_manager.cc:4841] TS 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843): tablet d950b2ee34c8424e9a2a633941c410ca (table test-table [id=18251fed664d4f6c9bb4481b7b3d6400]) successfully deleted
I20260619 15:02:50.622376 27322 ts_tablet_manager.cc:1918] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:02:50.622550 27322 log.cc:1196] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/d950b2ee34c8424e9a2a633941c410ca
I20260619 15:02:50.622711 27334 tablet_bootstrap.cc:492] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be: Bootstrap starting.
I20260619 15:02:50.623157 27322 ts_tablet_manager.cc:1939] T d950b2ee34c8424e9a2a633941c410ca P 0b7b4bb20a254493a141175daf679534: Deleting consensus metadata
I20260619 15:02:50.623874 27333 tablet_bootstrap.cc:492] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62: Bootstrap starting.
I20260619 15:02:50.624234 26718 catalog_manager.cc:4841] TS 0b7b4bb20a254493a141175daf679534 (127.24.103.65:32797): tablet d950b2ee34c8424e9a2a633941c410ca (table test-table [id=18251fed664d4f6c9bb4481b7b3d6400]) successfully deleted
I20260619 15:02:50.625011 27334 tablet_bootstrap.cc:654] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:50.622421 27335 tablet_bootstrap.cc:492] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534: Bootstrap starting.
I20260619 15:02:50.625700 27333 tablet_bootstrap.cc:654] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:50.626606 27335 tablet_bootstrap.cc:654] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:50.638330 27333 tablet_bootstrap.cc:492] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62: No bootstrap required, opened a new log
I20260619 15:02:50.638621 27333 ts_tablet_manager.cc:1397] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20260619 15:02:50.639238 27335 tablet_bootstrap.cc:492] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534: No bootstrap required, opened a new log
I20260619 15:02:50.639415 27335 ts_tablet_manager.cc:1397] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260619 15:02:50.639456 27334 tablet_bootstrap.cc:492] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be: No bootstrap required, opened a new log
I20260619 15:02:50.639667 27334 ts_tablet_manager.cc:1397] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
I20260619 15:02:50.639640 27333 raft_consensus.cc:348] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.639887 27333 raft_consensus.cc:374] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:50.639974 27333 raft_consensus.cc:729] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e781bf4699d49288db3eb6c3c47ee62, State: Initialized, Role: FOLLOWER
I20260619 15:02:50.640179 27335 raft_consensus.cc:348] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.640264 27333 consensus_queue.cc:260] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.640427 27335 raft_consensus.cc:374] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:50.640501 27335 raft_consensus.cc:729] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b4bb20a254493a141175daf679534, State: Initialized, Role: FOLLOWER
I20260619 15:02:50.640362 27334 raft_consensus.cc:348] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.640614 27334 raft_consensus.cc:374] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:50.640722 27334 raft_consensus.cc:729] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec1d74847e4425181613c9e261b21be, State: Initialized, Role: FOLLOWER
I20260619 15:02:50.640763 27335 consensus_queue.cc:260] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.640998 27333 ts_tablet_manager.cc:1428] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 15:02:50.640955 27334 consensus_queue.cc:260] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.641296 27335 ts_tablet_manager.cc:1428] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 15:02:50.641992 27334 ts_tablet_manager.cc:1428] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 15:02:50.879890 27299 raft_consensus.cc:482] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:02:50.880132 27299 raft_consensus.cc:504] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.880899 27299 leader_election.cc:288] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843), fec1d74847e4425181613c9e261b21be (127.24.103.67:35907)
I20260619 15:02:50.887234 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843dac42be5a49d2b0601b35202c9b21" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e781bf4699d49288db3eb6c3c47ee62" is_pre_election: true
I20260619 15:02:50.887526 27166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843dac42be5a49d2b0601b35202c9b21" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec1d74847e4425181613c9e261b21be" is_pre_election: true
I20260619 15:02:50.887583 27032 raft_consensus.cc:2454] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 0.
I20260619 15:02:50.887775 27166 raft_consensus.cc:2454] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 0.
I20260619 15:02:50.888123 26820 leader_election.cc:302] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b4bb20a254493a141175daf679534, 0e781bf4699d49288db3eb6c3c47ee62; no voters:
I20260619 15:02:50.888477 27299 raft_consensus.cc:2781] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:02:50.888590 27299 raft_consensus.cc:482] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:02:50.888675 27299 raft_consensus.cc:3037] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:50.891906 27299 raft_consensus.cc:504] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.892519 27299 leader_election.cc:288] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 election: Requested vote from peers 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843), fec1d74847e4425181613c9e261b21be (127.24.103.67:35907)
I20260619 15:02:50.893117 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843dac42be5a49d2b0601b35202c9b21" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e781bf4699d49288db3eb6c3c47ee62"
I20260619 15:02:50.893179 27166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843dac42be5a49d2b0601b35202c9b21" candidate_uuid: "0b7b4bb20a254493a141175daf679534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec1d74847e4425181613c9e261b21be"
I20260619 15:02:50.893342 27032 raft_consensus.cc:3037] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:50.893463 27166 raft_consensus.cc:3037] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:50.895507 27166 raft_consensus.cc:2454] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 1.
I20260619 15:02:50.895506 27032 raft_consensus.cc:2454] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b4bb20a254493a141175daf679534 in term 1.
I20260619 15:02:50.895980 26820 leader_election.cc:302] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b4bb20a254493a141175daf679534, 0e781bf4699d49288db3eb6c3c47ee62; no voters:
I20260619 15:02:50.896291 27299 raft_consensus.cc:2781] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:50.896448 27299 raft_consensus.cc:686] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 1 LEADER]: Becoming Leader. State: Replica: 0b7b4bb20a254493a141175daf679534, State: Running, Role: LEADER
I20260619 15:02:50.896726 27299 consensus_queue.cc:237] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } }
I20260619 15:02:50.899641 26731 catalog_manager.cc:5495] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b7b4bb20a254493a141175daf679534 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "0b7b4bb20a254493a141175daf679534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b4bb20a254493a141175daf679534" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 32797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 } health_report { overall_health: UNKNOWN } } }
I20260619 15:02:50.947444 27032 raft_consensus.cc:1261] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Refusing update from remote peer 0b7b4bb20a254493a141175daf679534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:50.947497 27166 raft_consensus.cc:1261] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Refusing update from remote peer 0b7b4bb20a254493a141175daf679534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:02:50.948032 27302 consensus_queue.cc:1035] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec1d74847e4425181613c9e261b21be" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:02:50.948355 27299 consensus_queue.cc:1035] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e781bf4699d49288db3eb6c3c47ee62" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 15:02:50.968389 27300 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 15:02:52.552238 26731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:02:52.552489 26731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:02:52.556752 26731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ae4cac1dd264b61b24758a57db42f5e: Sending DeleteTablet for 3 replicas of tablet 843dac42be5a49d2b0601b35202c9b21
I20260619 15:02:52.557631 26868 tablet_service.cc:1514] Processing DeleteTablet for tablet 843dac42be5a49d2b0601b35202c9b21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:52 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:59840
I20260619 15:02:52.557648 27008 tablet_service.cc:1514] Processing DeleteTablet for tablet 843dac42be5a49d2b0601b35202c9b21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:52 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:42588
I20260619 15:02:52.558269 27147 tablet_service.cc:1514] Processing DeleteTablet for tablet 843dac42be5a49d2b0601b35202c9b21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:02:52 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:55002
I20260619 15:02:52.558650 27371 tablet_replica.cc:330] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62: stopping tablet replica
I20260619 15:02:52.558970 27371 raft_consensus.cc:2229] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:02:52.559127 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 26803
I20260619 15:02:52.559295 27371 raft_consensus.cc:2258] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:52.560508 27371 ts_tablet_manager.cc:1905] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:52.570827 27371 ts_tablet_manager.cc:1918] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 15:02:52.571080 27371 log.cc:1196] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/843dac42be5a49d2b0601b35202c9b21
I20260619 15:02:52.571691 27371 ts_tablet_manager.cc:1939] T 843dac42be5a49d2b0601b35202c9b21 P 0e781bf4699d49288db3eb6c3c47ee62: Deleting consensus metadata
I20260619 15:02:52.573359 26717 catalog_manager.cc:4841] TS 0e781bf4699d49288db3eb6c3c47ee62 (127.24.103.66:34843): tablet 843dac42be5a49d2b0601b35202c9b21 (table test-table [id=ab043df153a34ca2a19b0e028a81af71]) successfully deleted
I20260619 15:02:52.573608 27373 tablet_replica.cc:330] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be: stopping tablet replica
I20260619 15:02:52.573807 27373 raft_consensus.cc:2229] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:02:52.574075 27373 raft_consensus.cc:2258] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:52.574658 27372 tablet_replica.cc:330] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534: stopping tablet replica
I20260619 15:02:52.574945 27372 raft_consensus.cc:2229] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:02:52.575300 27373 ts_tablet_manager.cc:1905] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:52.575325 27372 raft_consensus.cc:2258] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:02:52.576380 27372 ts_tablet_manager.cc:1905] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:02:52.580736 27373 ts_tablet_manager.cc:1918] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 15:02:52.580888 27373 log.cc:1196] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/843dac42be5a49d2b0601b35202c9b21
I20260619 15:02:52.581401 27373 ts_tablet_manager.cc:1939] T 843dac42be5a49d2b0601b35202c9b21 P fec1d74847e4425181613c9e261b21be: Deleting consensus metadata
I20260619 15:02:52.581979 27372 ts_tablet_manager.cc:1918] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 15:02:52.582132 27372 log.cc:1196] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/843dac42be5a49d2b0601b35202c9b21
I20260619 15:02:52.582659 27372 ts_tablet_manager.cc:1939] T 843dac42be5a49d2b0601b35202c9b21 P 0b7b4bb20a254493a141175daf679534: Deleting consensus metadata
I20260619 15:02:52.582644 26717 catalog_manager.cc:4841] TS fec1d74847e4425181613c9e261b21be (127.24.103.67:35907): tablet 843dac42be5a49d2b0601b35202c9b21 (table test-table [id=ab043df153a34ca2a19b0e028a81af71]) successfully deleted
I20260619 15:02:52.584029 26718 catalog_manager.cc:4841] TS 0b7b4bb20a254493a141175daf679534 (127.24.103.65:32797): tablet 843dac42be5a49d2b0601b35202c9b21 (table test-table [id=ab043df153a34ca2a19b0e028a81af71]) successfully deleted
Jun 19 15:02:52 dist-test-slave-zpbw krb5kdc[26687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:02:52.590845 27375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:52.562477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.24.103.65:32797)
0619 15:02:52.573787 (+ 11310us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:52.573797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:52.573829 (+ 32us) server_negotiation.cc:407] Connection header received
0619 15:02:52.573931 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:52.573939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:52.574067 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:52.575560 (+ 1493us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:52.576050 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:52.577152 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:52.578993 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:52.579678 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:52.579804 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:52.585484 (+ 5680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:52.585499 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:52.585516 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:52.585555 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:52.588260 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:52.588946 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:52.588953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:52.588956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:52.589074 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:52.589405 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:52.589408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:52.589410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:52.589761 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:52.589969 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:52.590313 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 15:02:52.590496 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10353,"thread_start_us":10564,"threads_started":1}
W20260619 15:02:52.591579 26934 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59884
W20260619 15:02:52.592443 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 26803: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:02:52.592638 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 26803
I20260619 15:02:52.618616 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 26943
Jun 19 15:02:52 dist-test-slave-zpbw krb5kdc[26687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:02:52.634248 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:52.619474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.24.103.66:34843)
0619 15:02:52.620092 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:52.620097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:52.620114 (+ 17us) server_negotiation.cc:407] Connection header received
0619 15:02:52.620167 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:52.620170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:52.620228 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:52.620325 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:52.621520 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:52.622336 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:52.623681 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:52.624095 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:52.624264 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:52.628512 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:52.628528 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:52.628545 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:52.628590 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:52.631391 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:52.632152 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:52.632160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:52.632162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:52.632297 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:52.632677 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:52.632683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:52.632686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:52.633045 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:52.633350 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:52.633659 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 15:02:52.633882 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":360,"threads_started":1}
W20260619 15:02:52.635039 27073 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42602
W20260619 15:02:52.635739 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 26943: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:02:52.635908 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 26943
I20260619 15:02:52.655916 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 27082
Jun 19 15:02:52 dist-test-slave-zpbw krb5kdc[26687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:02:52.672001 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:52.656892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.24.103.67:35907)
0619 15:02:52.657376 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:52.657380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:52.657399 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:02:52.657501 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:52.657505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:52.657570 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:52.657690 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:52.659083 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:52.659861 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:52.661125 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:52.661515 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:52.661619 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:52.666266 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:52.666283 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:52.666300 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:52.666345 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:52.669162 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:52.670003 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:52.670011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:52.670013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:52.670160 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:52.670543 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:52.670547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:52.670550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:52.671038 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:52.671256 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:52.671561 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 15:02:52.671734 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
W20260619 15:02:52.672716 27212 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55014
W20260619 15:02:52.673362 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 27082: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:02:52.673521 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 27082
I20260619 15:02:52.695369 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 26700
I20260619 15:02:52.696632 26772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:02:52.812270 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 26700
2026-06-19T15:02:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8474 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:02:52 dist-test-slave-zpbw krb5kdc[27381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:02:52 dist-test-slave-zpbw krb5kdc[27381](info): set up 2 sockets
Jun 19 15:02:52 dist-test-slave-zpbw krb5kdc[27381](info): commencing operation
krb5kdc: starting...
W20260619 15:02:53.907313 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:02:54 dist-test-slave-zpbw krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:02:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:02:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:02:54.094766 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34877
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:34877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:54.411512 27394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:54.411811 27394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:54.411976 27394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:54.419209 27394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:02:54.419320 27394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:54.419425 27394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:54.419529 27394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:02:54.419615 27394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:02:54.429802 27394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:34877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:54.431258 27394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:54.432615 27394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:54.444427 27400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:54.444557 27402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:54.444533 27399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:54.445806 27394 server_base.cc:1034] running on GCE node
I20260619 15:02:54.446908 27394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:54.448341 27394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:54.449569 27394 hybrid_clock.cc:648] HybridClock initialized: now 1781881374449516 us; error 59 us; skew 500 ppm
I20260619 15:02:54.449921 27394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:54 dist-test-slave-zpbw krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881374, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:54.453037 27394 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:02:54.454607 27394 webserver.cc:460] Webserver started at http://127.24.103.126:40787/ using document root <none> and password file <none>
I20260619 15:02:54.455371 27394 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:54.455507 27394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:54.455818 27394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:54.458767 27394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "a2300ca64075449a82c9133210b8fd3d"
format_stamp: "Formatted at 2026-06-19 15:02:54 on dist-test-slave-zpbw"
server_key: "76fb2a32866bad1ef18505b0a32081ae"
server_key_iv: "c2a0a087d061e9e83b5f650128095957"
server_key_version: "encryptionkey@0"
I20260619 15:02:54.459643 27394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "a2300ca64075449a82c9133210b8fd3d"
format_stamp: "Formatted at 2026-06-19 15:02:54 on dist-test-slave-zpbw"
server_key: "76fb2a32866bad1ef18505b0a32081ae"
server_key_iv: "c2a0a087d061e9e83b5f650128095957"
server_key_version: "encryptionkey@0"
I20260619 15:02:54.464972 27394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 15:02:54.468611 27409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:54.470033 27394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 15:02:54.470230 27394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "a2300ca64075449a82c9133210b8fd3d"
format_stamp: "Formatted at 2026-06-19 15:02:54 on dist-test-slave-zpbw"
server_key: "76fb2a32866bad1ef18505b0a32081ae"
server_key_iv: "c2a0a087d061e9e83b5f650128095957"
server_key_version: "encryptionkey@0"
I20260619 15:02:54.470446 27394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:54.501492 27394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:54.512533 27394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:54.512913 27394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:54.532816 27394 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:34877
I20260619 15:02:54.532788 27460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:34877 every 8 connection(s)
I20260619 15:02:54.534337 27394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:02:54.538256 27461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:54.543331 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 27394
I20260619 15:02:54.543573 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:02:54.544247 24989 external_mini_cluster.cc:1428] Setting key 5cd10018ac418734dbaf2f9a890aab84
I20260619 15:02:54.550266 27461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d: Bootstrap starting.
I20260619 15:02:54.554030 27461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:54.555297 27461 log.cc:824] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:54.558498 27461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d: No bootstrap required, opened a new log
Jun 19 15:02:54 dist-test-slave-zpbw krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:54.568651 27461 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2300ca64075449a82c9133210b8fd3d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34877 } }
I20260619 15:02:54.569221 27461 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:54.569337 27461 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2300ca64075449a82c9133210b8fd3d, State: Initialized, Role: FOLLOWER
I20260619 15:02:54.570050 27461 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2300ca64075449a82c9133210b8fd3d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34877 } }
I20260619 15:02:54.570371 27461 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:02:54.570518 27461 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:02:54.570716 27461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:54.571096 27464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:54.546474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.24.103.126:34877)
0619 15:02:54.547427 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:54.547443 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:54.547503 (+ 60us) server_negotiation.cc:407] Connection header received
0619 15:02:54.549857 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:54.549885 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:54.550420 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:54.550907 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:54.552420 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:54.554479 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:54.556192 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:54.557089 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:54.557292 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:54.561604 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:54.561615 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:54.561635 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:54.561686 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:54.566158 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:54.566972 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:54.566981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:54.566991 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:54.567229 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:54.567581 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:54.567586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:54.567589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:54.568316 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:54.568604 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:54.568944 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 15:02:54.569510 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":456,"threads_started":1}
I20260619 15:02:54.573295 27461 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2300ca64075449a82c9133210b8fd3d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34877 } }
I20260619 15:02:54.573891 27461 leader_election.cc:302] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2300ca64075449a82c9133210b8fd3d; no voters:
I20260619 15:02:54.574501 27461 leader_election.cc:288] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:02:54.574623 27466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:54.575604 27466 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [term 1 LEADER]: Becoming Leader. State: Replica: a2300ca64075449a82c9133210b8fd3d, State: Running, Role: LEADER
I20260619 15:02:54.576082 27466 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2300ca64075449a82c9133210b8fd3d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34877 } }
I20260619 15:02:54.576880 27461 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:02:54.578922 27468 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2300ca64075449a82c9133210b8fd3d. Latest consensus state: current_term: 1 leader_uuid: "a2300ca64075449a82c9133210b8fd3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2300ca64075449a82c9133210b8fd3d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34877 } } }
I20260619 15:02:54.579358 27468 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [sys.catalog]: This master's current role is: LEADER
I20260619 15:02:54.584641 27467 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2300ca64075449a82c9133210b8fd3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2300ca64075449a82c9133210b8fd3d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34877 } } }
I20260619 15:02:54.584939 27467 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d [sys.catalog]: This master's current role is: LEADER
W20260619 15:02:54.588791 27482 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:02:54.588958 27482 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:02:54.589119 27475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:02:54.590937 27475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:02:54.598932 27475 catalog_manager.cc:1347] Generated new cluster ID: 64699c60312a4ab0b6e0baba685b9b21
I20260619 15:02:54.599064 27475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:02:54.611249 27475 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:02:54.612615 27475 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:02:54.629149 27475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2300ca64075449a82c9133210b8fd3d: Generated new TSK 0
I20260619 15:02:54.630329 27475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:02:54.688301 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34877
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:55.020943 27487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:55.021246 27487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:55.021394 27487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:55.029228 27487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:55.029359 27487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:55.029592 27487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:02:55.039916 27487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:55.041303 27487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:55.042665 27487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:55.056042 27492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:55.055969 27493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:55.055974 27495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:55.056589 27487 server_base.cc:1034] running on GCE node
I20260619 15:02:55.057407 27487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:55.058408 27487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:55.059666 27487 hybrid_clock.cc:648] HybridClock initialized: now 1781881375059594 us; error 64 us; skew 500 ppm
I20260619 15:02:55.060025 27487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:55 dist-test-slave-zpbw krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881375, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:55.063419 27487 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:02:55.065083 27487 webserver.cc:460] Webserver started at http://127.24.103.65:38623/ using document root <none> and password file <none>
I20260619 15:02:55.065827 27487 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:55.065970 27487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:55.066288 27487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:55.069336 27487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "631e4e7c6d2d4a5fb73496137c6a8800"
format_stamp: "Formatted at 2026-06-19 15:02:55 on dist-test-slave-zpbw"
server_key: "35c44ad3c105c9793407a9dc195e12f4"
server_key_iv: "a4b11c8ba41da8fd6401c05ef46e814e"
server_key_version: "encryptionkey@0"
I20260619 15:02:55.070184 27487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "631e4e7c6d2d4a5fb73496137c6a8800"
format_stamp: "Formatted at 2026-06-19 15:02:55 on dist-test-slave-zpbw"
server_key: "35c44ad3c105c9793407a9dc195e12f4"
server_key_iv: "a4b11c8ba41da8fd6401c05ef46e814e"
server_key_version: "encryptionkey@0"
I20260619 15:02:55.075711 27487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 15:02:55.079245 27502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:55.080735 27487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 15:02:55.080922 27487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "631e4e7c6d2d4a5fb73496137c6a8800"
format_stamp: "Formatted at 2026-06-19 15:02:55 on dist-test-slave-zpbw"
server_key: "35c44ad3c105c9793407a9dc195e12f4"
server_key_iv: "a4b11c8ba41da8fd6401c05ef46e814e"
server_key_version: "encryptionkey@0"
I20260619 15:02:55.081130 27487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:55.094612 27487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:55.098523 27487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:55.098884 27487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:55.099846 27487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:02:55.101138 27487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:55.101236 27487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:55.101370 27487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:55.101444 27487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:55.135311 27487 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:43319
I20260619 15:02:55.135349 27614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:43319 every 8 connection(s)
I20260619 15:02:55.136626 27487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:02:55.139519 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 27487
I20260619 15:02:55.139745 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:02:55.140601 24989 external_mini_cluster.cc:1428] Setting key 1fee60f9eb2fe3531e2d83f6337438de
Jun 19 15:02:55 dist-test-slave-zpbw krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881375, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:55.158475 27617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:55.139129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:46333 (local address 127.24.103.126:34877)
0619 15:02:55.139615 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:55.139621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:55.140790 (+ 1169us) server_negotiation.cc:407] Connection header received
0619 15:02:55.142758 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:55.142763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:55.142869 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:55.142981 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:55.144503 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:55.145238 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:55.146834 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:55.147242 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:55.147378 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:55.151528 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:55.151537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:55.151545 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:55.151591 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:55.155297 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:55.156077 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:55.156085 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:55.156092 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:55.156266 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:55.156719 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:55.156724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:55.156727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:55.156998 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:55.157145 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:55.157871 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 15:02:55.158106 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":243,"threads_started":1}
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260619 15:02:55.159610 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:55.139847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34877 (local address 127.24.103.65:46333)
0619 15:02:55.140587 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:55.140624 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:55.142482 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:55.143228 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:55.143237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:55.143764 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:55.144337 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:55.144354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:55.145395 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:55.145399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:55.146651 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:55.146663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:55.147471 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:55.147480 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:55.147776 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:55.148538 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:55.148572 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:55.151291 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:55.155461 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:55.155470 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:55.155485 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:55.155948 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:55.156389 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:55.156392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:55.156395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:55.156597 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:55.157141 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:55.157150 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:55.157510 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 15:02:55.157777 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 15:02:55.158349 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":303,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
I20260619 15:02:55.161181 27615 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34877
I20260619 15:02:55.161501 27615 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:55.162247 27615 heartbeater.cc:511] Master 127.24.103.126:34877 requested a full tablet report, sending...
I20260619 15:02:55.164402 27426 ts_manager.cc:194] Registered new tserver with Master: 631e4e7c6d2d4a5fb73496137c6a8800 (127.24.103.65:43319)
I20260619 15:02:55.165879 27426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:46333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:02:55.183368 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34877
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:55.506752 27621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:55.507069 27621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:55.507169 27621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:55.514559 27621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:55.514683 27621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:55.514858 27621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:02:55.524763 27621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:55.526031 27621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:55.527371 27621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:55.542507 27626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:55.543174 27627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:55.543334 27629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:55.543675 27621 server_base.cc:1034] running on GCE node
I20260619 15:02:55.544360 27621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:55.545361 27621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:55.546597 27621 hybrid_clock.cc:648] HybridClock initialized: now 1781881375546535 us; error 53 us; skew 500 ppm
I20260619 15:02:55.546974 27621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:55 dist-test-slave-zpbw krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881375, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:55.550148 27621 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:02:55.551815 27621 webserver.cc:460] Webserver started at http://127.24.103.66:38605/ using document root <none> and password file <none>
I20260619 15:02:55.552516 27621 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:55.552644 27621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:55.552969 27621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:55.556008 27621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "1d1f788da3384deaa4256c9a58b2cec8"
format_stamp: "Formatted at 2026-06-19 15:02:55 on dist-test-slave-zpbw"
server_key: "235d4442debf54c45d35c86442025d2f"
server_key_iv: "a3f05db2699fb4b3d4cca9f9ee6a7298"
server_key_version: "encryptionkey@0"
I20260619 15:02:55.556893 27621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "1d1f788da3384deaa4256c9a58b2cec8"
format_stamp: "Formatted at 2026-06-19 15:02:55 on dist-test-slave-zpbw"
server_key: "235d4442debf54c45d35c86442025d2f"
server_key_iv: "a3f05db2699fb4b3d4cca9f9ee6a7298"
server_key_version: "encryptionkey@0"
I20260619 15:02:55.562366 27621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:02:55.565986 27636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:55.567456 27621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:02:55.567608 27621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "1d1f788da3384deaa4256c9a58b2cec8"
format_stamp: "Formatted at 2026-06-19 15:02:55 on dist-test-slave-zpbw"
server_key: "235d4442debf54c45d35c86442025d2f"
server_key_iv: "a3f05db2699fb4b3d4cca9f9ee6a7298"
server_key_version: "encryptionkey@0"
I20260619 15:02:55.567839 27621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:55.584435 27621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:55.589149 27621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:55.589452 27621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:55.590416 27621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:02:55.592032 27621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:55.592147 27621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:55.592280 27621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:55.592350 27621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:55.625029 27621 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:46801
I20260619 15:02:55.625068 27748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:46801 every 8 connection(s)
I20260619 15:02:55.626418 27621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:02:55.632330 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 27621
I20260619 15:02:55.632583 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:02:55.633018 24989 external_mini_cluster.cc:1428] Setting key 09776e68f4957eee771fe24e68287705
Jun 19 15:02:55 dist-test-slave-zpbw krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881375, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:55.646651 27617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:55.629060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:40633 (local address 127.24.103.126:34877)
0619 15:02:55.629243 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:55.629248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:55.630475 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 15:02:55.632401 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:55.632407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:55.632512 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:55.632620 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:55.634284 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:55.635152 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:55.636697 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:55.637111 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:55.637242 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:55.640830 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:55.640835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:55.640838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:55.640876 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:55.643584 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:55.644324 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:55.644330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:55.644333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:55.644454 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:55.644880 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:55.644886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:55.644889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:55.645168 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:55.645323 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:55.646049 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 15:02:55.646299 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:02:55.647676 27751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:55.629595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34877 (local address 127.24.103.66:40633)
0619 15:02:55.630303 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:55.630339 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:55.632121 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:55.632867 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:55.632876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:55.633443 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:55.634123 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:55.634141 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:55.635337 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:55.635341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:55.636554 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:55.636565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:55.637329 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:55.637336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:55.637597 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:55.638281 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:55.638304 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:55.640512 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:55.643740 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:55.643749 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:55.643768 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:55.644204 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:55.644560 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:55.644565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:55.644569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:55.644773 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:55.645299 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:55.645308 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:55.645673 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 15:02:55.645962 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 15:02:55.646517 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":266,"threads_started":1}
I20260619 15:02:55.649354 27749 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34877
I20260619 15:02:55.649658 27749 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260619 15:02:55.650375 27749 heartbeater.cc:511] Master 127.24.103.126:34877 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
I20260619 15:02:55.652228 27426 ts_manager.cc:194] Registered new tserver with Master: 1d1f788da3384deaa4256c9a58b2cec8 (127.24.103.66:46801)
I20260619 15:02:55.653443 27426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:40633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:02:55.673923 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34877
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:56.001475 27754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:56.001780 27754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:56.001878 27754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:56.009053 27754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:56.009177 27754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:56.009290 27754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:02:56.019297 27754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:56.020534 27754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:56.021903 27754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:56.034961 27759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:56.034989 27760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:56.034917 27762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:56.036217 27754 server_base.cc:1034] running on GCE node
I20260619 15:02:56.036826 27754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:56.037787 27754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:56.038983 27754 hybrid_clock.cc:648] HybridClock initialized: now 1781881376038921 us; error 57 us; skew 500 ppm
I20260619 15:02:56.039311 27754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:56 dist-test-slave-zpbw krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881376, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:56.042572 27754 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:02:56.044116 27754 webserver.cc:460] Webserver started at http://127.24.103.67:45031/ using document root <none> and password file <none>
I20260619 15:02:56.044792 27754 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:56.044891 27754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:56.045156 27754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:56.048511 27754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "02b6a1943aef413fb6a16e511d9be297"
format_stamp: "Formatted at 2026-06-19 15:02:56 on dist-test-slave-zpbw"
server_key: "4b40733bd3aab03e96058b5aa25ba821"
server_key_iv: "cb93b79f419f8142bfc638b44b2f69fe"
server_key_version: "encryptionkey@0"
I20260619 15:02:56.049319 27754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "02b6a1943aef413fb6a16e511d9be297"
format_stamp: "Formatted at 2026-06-19 15:02:56 on dist-test-slave-zpbw"
server_key: "4b40733bd3aab03e96058b5aa25ba821"
server_key_iv: "cb93b79f419f8142bfc638b44b2f69fe"
server_key_version: "encryptionkey@0"
I20260619 15:02:56.054695 27754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 15:02:56.058408 27769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:56.059859 27754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:02:56.060032 27754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "02b6a1943aef413fb6a16e511d9be297"
format_stamp: "Formatted at 2026-06-19 15:02:56 on dist-test-slave-zpbw"
server_key: "4b40733bd3aab03e96058b5aa25ba821"
server_key_iv: "cb93b79f419f8142bfc638b44b2f69fe"
server_key_version: "encryptionkey@0"
I20260619 15:02:56.060235 27754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:56.082870 27754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:56.087422 27754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:56.087713 27754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:56.088721 27754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:02:56.090008 27754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:02:56.090081 27754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:56.090240 27754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:02:56.090291 27754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:56.122598 27754 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:39507
I20260619 15:02:56.122704 27881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:39507 every 8 connection(s)
I20260619 15:02:56.123943 27754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:02:56.133704 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 27754
I20260619 15:02:56.133925 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:02:56.134341 24989 external_mini_cluster.cc:1428] Setting key 616a5911f9809a14bc2fa1708871820b
Jun 19 15:02:56 dist-test-slave-zpbw krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881376, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:56.144977 27617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:56.126596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:39021 (local address 127.24.103.126:34877)
0619 15:02:56.126846 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:56.126857 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:56.128120 (+ 1263us) server_negotiation.cc:407] Connection header received
0619 15:02:56.129985 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:56.129992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:56.130086 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:56.130196 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:56.131789 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:56.132669 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:56.134310 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:56.134921 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:56.135062 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:56.139254 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:56.139260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:56.139265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:56.139306 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:56.142234 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:56.142980 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:56.142984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:56.142986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:56.143112 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:56.143482 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:56.143485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:56.143487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:56.143772 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:56.143879 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:56.144573 (+ 694us) server_negotiation.cc:299] Negotiation successful
0619 15:02:56.144741 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:02:56.145972 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:56.127196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34877 (local address 127.24.103.67:39021)
0619 15:02:56.127955 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0619 15:02:56.128006 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 15:02:56.129705 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:02:56.130419 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:02:56.130427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:02:56.131044 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:02:56.131620 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:56.131640 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:56.132816 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:56.132820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:56.134169 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:02:56.134190 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:02:56.135171 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:02:56.135179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:02:56.135505 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:56.136371 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:02:56.136404 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:02:56.139033 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:02:56.142362 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:56.142369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:56.142380 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:56.142858 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:56.143210 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:02:56.143213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:02:56.143215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:02:56.143388 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:02:56.143888 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:02:56.143894 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:02:56.144220 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 15:02:56.144443 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 15:02:56.144821 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":277,"threads_started":1}
I20260619 15:02:56.147353 27882 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34877
I20260619 15:02:56.147644 27882 heartbeater.cc:464] Registering TS with master...
I20260619 15:02:56.148309 27882 heartbeater.cc:511] Master 127.24.103.126:34877 requested a full tablet report, sending...
I20260619 15:02:56.149837 27426 ts_manager.cc:194] Registered new tserver with Master: 02b6a1943aef413fb6a16e511d9be297 (127.24.103.67:39507)
I20260619 15:02:56.150578 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:02:56.150853 27426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:39021
I20260619 15:02:56.168936 27615 heartbeater.cc:503] Master 127.24.103.126:34877 was elected leader, sending a full tablet report...
W20260619 15:02:56.181742 27617 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:02:56.179104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.24.103.126:34877)
0619 15:02:56.179291 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:56.179295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:56.179841 (+ 546us) server_negotiation.cc:407] Connection header received
0619 15:02:56.180056 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:56.180061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:56.180129 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:56.180237 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:56.181466 (+ 1229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40252: BlockingRecv error: recv got EOF from 127.0.0.1:40252 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:02:56.184867 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 27487
W20260619 15:02:56.187748 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 27487: Not authorized: Client connection negotiation failed: client connection to 127.24.103.65:43319: server requires authentication, but client does not have Kerberos credentials available
I20260619 15:02:56.187959 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 27487
I20260619 15:02:56.203804 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 27621
W20260619 15:02:56.206548 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 27621: Not authorized: Client connection negotiation failed: client connection to 127.24.103.66:46801: server requires authentication, but client does not have Kerberos credentials available
I20260619 15:02:56.206738 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 27621
I20260619 15:02:56.222220 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 27754
W20260619 15:02:56.225237 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 27754: Not authorized: Client connection negotiation failed: client connection to 127.24.103.67:39507: server requires authentication, but client does not have Kerberos credentials available
I20260619 15:02:56.225407 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 27754
I20260619 15:02:56.240664 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 27394
I20260619 15:02:56.242007 27456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:02:56.347177 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 27394
2026-06-19T15:02:56Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3531 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:02:56 dist-test-slave-zpbw krb5kdc[27899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:02:56 dist-test-slave-zpbw krb5kdc[27899](info): set up 2 sockets
Jun 19 15:02:56 dist-test-slave-zpbw krb5kdc[27899](info): commencing operation
krb5kdc: starting...
W20260619 15:02:57.443326 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:02:57 dist-test-slave-zpbw krb5kdc[27899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:02:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:02:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:02:57.636364 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39733
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:39733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:57.975792 27912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:57.976191 27912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:57.976387 27912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:57.984066 27912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:02:57.984238 27912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:57.984360 27912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:57.984489 27912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:02:57.984587 27912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:02:57.995307 27912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:39733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:57.996778 27912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:57.998217 27912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:58.011332 27920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:58.011364 27918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:58.011621 27917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:58.012048 27912 server_base.cc:1034] running on GCE node
I20260619 15:02:58.013208 27912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:58.014899 27912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:58.016126 27912 hybrid_clock.cc:648] HybridClock initialized: now 1781881378016074 us; error 52 us; skew 500 ppm
I20260619 15:02:58.016491 27912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:58 dist-test-slave-zpbw krb5kdc[27899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881378, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:58.019611 27912 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:02:58.021152 27912 webserver.cc:460] Webserver started at http://127.24.103.126:37117/ using document root <none> and password file <none>
I20260619 15:02:58.021801 27912 fs_manager.cc:359] Metadata directory not provided
I20260619 15:02:58.021904 27912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:02:58.022178 27912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:02:58.025367 27912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "4fad13366d994e9ab31de9a405150b00"
format_stamp: "Formatted at 2026-06-19 15:02:58 on dist-test-slave-zpbw"
server_key: "88e66c9bfef29bf04618e149b761a2e2"
server_key_iv: "9d3b28e61ec803a9851c95fdf18fa223"
server_key_version: "encryptionkey@0"
I20260619 15:02:58.026129 27912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "4fad13366d994e9ab31de9a405150b00"
format_stamp: "Formatted at 2026-06-19 15:02:58 on dist-test-slave-zpbw"
server_key: "88e66c9bfef29bf04618e149b761a2e2"
server_key_iv: "9d3b28e61ec803a9851c95fdf18fa223"
server_key_version: "encryptionkey@0"
I20260619 15:02:58.031503 27912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:02:58.035261 27927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:02:58.036718 27912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:02:58.036909 27912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "4fad13366d994e9ab31de9a405150b00"
format_stamp: "Formatted at 2026-06-19 15:02:58 on dist-test-slave-zpbw"
server_key: "88e66c9bfef29bf04618e149b761a2e2"
server_key_iv: "9d3b28e61ec803a9851c95fdf18fa223"
server_key_version: "encryptionkey@0"
I20260619 15:02:58.037137 27912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:02:58.066107 27912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:02:58.070420 27912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:02:58.070730 27912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:02:58.090047 27912 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:39733
I20260619 15:02:58.090082 27978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:39733 every 8 connection(s)
I20260619 15:02:58.091550 27912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:02:58.095525 27979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:02:58.096468 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 27912
I20260619 15:02:58.096711 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:02:58.097560 24989 external_mini_cluster.cc:1428] Setting key a2cc46b1d4d8b1da6c32cb639d4b88c8
I20260619 15:02:58.110808 27979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00: Bootstrap starting.
Jun 19 15:02:58 dist-test-slave-zpbw krb5kdc[27899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:02:58.114470 27979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00: Neither blocks nor log segments found. Creating new log.
I20260619 15:02:58.115841 27979 log.cc:824] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00: Log is configured to *not* fsync() on all Append() calls
I20260619 15:02:58.118866 27979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00: No bootstrap required, opened a new log
I20260619 15:02:58.121306 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:58.099577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.24.103.126:39733)
0619 15:02:58.100419 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:58.100430 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:58.100477 (+ 47us) server_negotiation.cc:407] Connection header received
0619 15:02:58.102131 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:58.102154 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:58.102579 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:58.103014 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:58.104908 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:58.106208 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:58.108095 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:58.108874 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:58.109024 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:58.113682 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:58.113691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:58.113705 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:02:58.113749 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:58.117059 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:58.117822 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:58.117830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:58.117836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:58.117998 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:02:58.118392 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:02:58.118401 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:02:58.118404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:02:58.119062 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:58.119398 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:58.119814 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 15:02:58.120267 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":382,"threads_started":1}
I20260619 15:02:58.125669 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:02:58.127674 27979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fad13366d994e9ab31de9a405150b00" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39733 } }
I20260619 15:02:58.128062 27979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:02:58.128196 27979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fad13366d994e9ab31de9a405150b00, State: Initialized, Role: FOLLOWER
I20260619 15:02:58.128834 27979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fad13366d994e9ab31de9a405150b00" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39733 } }
I20260619 15:02:58.129058 27979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:02:58.129195 27979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:02:58.129369 27979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:02:58.131623 27979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fad13366d994e9ab31de9a405150b00" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39733 } }
I20260619 15:02:58.132138 27979 leader_election.cc:302] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fad13366d994e9ab31de9a405150b00; no voters:
I20260619 15:02:58.132790 27979 leader_election.cc:288] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:02:58.132910 27985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:02:58.134397 27985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [term 1 LEADER]: Becoming Leader. State: Replica: 4fad13366d994e9ab31de9a405150b00, State: Running, Role: LEADER
I20260619 15:02:58.135046 27985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fad13366d994e9ab31de9a405150b00" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39733 } }
I20260619 15:02:58.135206 27979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:02:58.137944 27986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fad13366d994e9ab31de9a405150b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fad13366d994e9ab31de9a405150b00" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39733 } } }
I20260619 15:02:58.138322 27986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [sys.catalog]: This master's current role is: LEADER
I20260619 15:02:58.138875 27987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fad13366d994e9ab31de9a405150b00. Latest consensus state: current_term: 1 leader_uuid: "4fad13366d994e9ab31de9a405150b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fad13366d994e9ab31de9a405150b00" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39733 } } }
I20260619 15:02:58.139159 27987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00 [sys.catalog]: This master's current role is: LEADER
I20260619 15:02:58.141620 27993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:02:58.147415 27993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:02:58.167580 27993 catalog_manager.cc:1347] Generated new cluster ID: bb8b844426d048e09dd59c57a3607ba7
I20260619 15:02:58.167706 27993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:02:58.185734 27993 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:02:58.187409 27993 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:02:58.196529 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:02:58.185561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.24.103.126:39733)
0619 15:02:58.185745 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 15:02:58.185753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:02:58.187427 (+ 1674us) server_negotiation.cc:407] Connection header received
0619 15:02:58.187644 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:02:58.187650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:02:58.187734 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:02:58.187812 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:02:58.189448 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:58.190604 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:58.192449 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:02:58.193214 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:02:58.193390 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:02:58.194609 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:02:58.194618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:02:58.194623 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:02:58.195612 (+ 989us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:02:58.195695 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:02:58.195741 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:02:58.195870 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 15:02:58.196015 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 15:02:58.196121 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 15:02:58.201932 27993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fad13366d994e9ab31de9a405150b00: Generated new TSK 0
I20260619 15:02:58.203161 27993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:02:58.217640 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 27912
I20260619 15:02:58.219154 27974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:02:58.310271 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 27912
2026-06-19T15:02:58Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1956 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:02:58 dist-test-slave-zpbw krb5kdc[28013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:02:58 dist-test-slave-zpbw krb5kdc[28013](info): set up 2 sockets
Jun 19 15:02:58 dist-test-slave-zpbw krb5kdc[28013](info): commencing operation
krb5kdc: starting...
W20260619 15:02:59.399423 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:02:59 dist-test-slave-zpbw krb5kdc[28013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:02:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:02:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:02:59.610345 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38367
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:02:59.953464 28026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:02:59.953871 28026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:02:59.954037 28026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:02:59.961617 28026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:02:59.961807 28026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:02:59.961930 28026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:02:59.962035 28026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:02:59.962116 28026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:02:59.972497 28026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:02:59.974062 28026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:02:59.975785 28026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:02:59.988557 28032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:59.988603 28034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:02:59.988657 28031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:02:59.989539 28026 server_base.cc:1034] running on GCE node
I20260619 15:02:59.990411 28026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:02:59.992049 28026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:02:59.993320 28026 hybrid_clock.cc:648] HybridClock initialized: now 1781881379993278 us; error 62 us; skew 500 ppm
I20260619 15:02:59.993809 28026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:02:59 dist-test-slave-zpbw krb5kdc[28013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881379, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:02:59.997327 28026 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:02:59.999068 28026 webserver.cc:460] Webserver started at http://127.24.103.126:42765/ using document root <none> and password file <none>
I20260619 15:02:59.999912 28026 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:00.000068 28026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:00.000396 28026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:00.003692 28026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "540d0475ee1545bd998c7fa48381f799"
format_stamp: "Formatted at 2026-06-19 15:02:59 on dist-test-slave-zpbw"
server_key: "7c6204cf190517e5878badc2503bc9f5"
server_key_iv: "80c88577483cc6a2977b6dacac6952a6"
server_key_version: "encryptionkey@0"
I20260619 15:03:00.004592 28026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "540d0475ee1545bd998c7fa48381f799"
format_stamp: "Formatted at 2026-06-19 15:02:59 on dist-test-slave-zpbw"
server_key: "7c6204cf190517e5878badc2503bc9f5"
server_key_iv: "80c88577483cc6a2977b6dacac6952a6"
server_key_version: "encryptionkey@0"
I20260619 15:03:00.010318 28026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 15:03:00.014186 28041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:00.015793 28026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:03:00.016038 28026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "540d0475ee1545bd998c7fa48381f799"
format_stamp: "Formatted at 2026-06-19 15:02:59 on dist-test-slave-zpbw"
server_key: "7c6204cf190517e5878badc2503bc9f5"
server_key_iv: "80c88577483cc6a2977b6dacac6952a6"
server_key_version: "encryptionkey@0"
I20260619 15:03:00.016306 28026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:00.031148 28026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:00.035883 28026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:00.036227 28026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:00.056727 28026 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:38367
I20260619 15:03:00.056757 28092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:38367 every 8 connection(s)
I20260619 15:03:00.058225 28026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:00.061312 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 28026
I20260619 15:03:00.061546 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:03:00.062188 24989 external_mini_cluster.cc:1428] Setting key 56482ee5332f3dcfada187e87a11e3df
I20260619 15:03:00.062232 28093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:00.077720 28093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799: Bootstrap starting.
Jun 19 15:03:00 dist-test-slave-zpbw krb5kdc[28013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:00.081110 28093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:00.082252 28093 log.cc:824] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:00.085363 28093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799: No bootstrap required, opened a new log
I20260619 15:03:00.086951 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:00.064428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.24.103.126:38367)
0619 15:03:00.065331 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:00.065347 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:00.065390 (+ 43us) server_negotiation.cc:407] Connection header received
0619 15:03:00.067161 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:00.067188 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:00.067651 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:00.068052 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:00.069884 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:00.071297 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:00.073130 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:00.073897 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:00.074061 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:00.079192 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:00.079201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:00.079219 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:00.079264 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:00.082591 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:00.083418 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:00.083436 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:00.083443 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:00.083620 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:00.084039 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:00.084046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:00.084049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:00.084628 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:00.084975 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:00.085424 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 15:03:00.085876 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":394,"threads_started":1}
I20260619 15:03:00.094209 28093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540d0475ee1545bd998c7fa48381f799" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38367 } }
I20260619 15:03:00.094645 28093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:00.094764 28093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540d0475ee1545bd998c7fa48381f799, State: Initialized, Role: FOLLOWER
I20260619 15:03:00.095517 28093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540d0475ee1545bd998c7fa48381f799" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38367 } }
I20260619 15:03:00.095806 28093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:00.095968 28093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:00.096153 28093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:00.098342 28093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540d0475ee1545bd998c7fa48381f799" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38367 } }
I20260619 15:03:00.098964 28093 leader_election.cc:302] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 540d0475ee1545bd998c7fa48381f799; no voters:
I20260619 15:03:00.099594 28093 leader_election.cc:288] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:03:00.099778 28098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:00.100570 28098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [term 1 LEADER]: Becoming Leader. State: Replica: 540d0475ee1545bd998c7fa48381f799, State: Running, Role: LEADER
I20260619 15:03:00.101239 28098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540d0475ee1545bd998c7fa48381f799" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38367 } }
I20260619 15:03:00.101816 28093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:00.105057 28099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "540d0475ee1545bd998c7fa48381f799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540d0475ee1545bd998c7fa48381f799" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38367 } } }
I20260619 15:03:00.105469 28099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:00.105008 28100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 540d0475ee1545bd998c7fa48381f799. Latest consensus state: current_term: 1 leader_uuid: "540d0475ee1545bd998c7fa48381f799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540d0475ee1545bd998c7fa48381f799" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38367 } } }
I20260619 15:03:00.105708 28100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:00.107846 28106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:00.113368 28106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:00.123967 28106 catalog_manager.cc:1347] Generated new cluster ID: dc4d9c5de0134cd9b15af4e3f211ddfb
I20260619 15:03:00.124132 28106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:00.162592 28106 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:03:00.164021 28106 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:00.175391 28106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 540d0475ee1545bd998c7fa48381f799: Generated new TSK 0
I20260619 15:03:00.176326 28106 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:03:00.230633 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38367
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:00.568005 28119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:00.568302 28119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:00.568395 28119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:00.575824 28119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:00.575958 28119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:00.576076 28119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:03:00.586833 28119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:00.588219 28119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:00.589625 28119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:00.603420 28127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:00.603437 28125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:00.603969 28124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:00.604912 28119 server_base.cc:1034] running on GCE node
I20260619 15:03:00.605576 28119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:00.606570 28119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:00.607810 28119 hybrid_clock.cc:648] HybridClock initialized: now 1781881380607737 us; error 69 us; skew 500 ppm
I20260619 15:03:00.608206 28119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:00 dist-test-slave-zpbw krb5kdc[28013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881380, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:00.611335 28119 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:03:00.612792 28119 webserver.cc:460] Webserver started at http://127.24.103.65:39645/ using document root <none> and password file <none>
I20260619 15:03:00.613503 28119 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:00.613646 28119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:00.613943 28119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:00.616990 28119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "ade8cd5630504d728cb1bc41c79d0361"
format_stamp: "Formatted at 2026-06-19 15:03:00 on dist-test-slave-zpbw"
server_key: "1ddea50a78c060aed5df770ee868b1c1"
server_key_iv: "7106b87c03cf9aa23f866a5649636c9f"
server_key_version: "encryptionkey@0"
I20260619 15:03:00.617792 28119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "ade8cd5630504d728cb1bc41c79d0361"
format_stamp: "Formatted at 2026-06-19 15:03:00 on dist-test-slave-zpbw"
server_key: "1ddea50a78c060aed5df770ee868b1c1"
server_key_iv: "7106b87c03cf9aa23f866a5649636c9f"
server_key_version: "encryptionkey@0"
I20260619 15:03:00.623204 28119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260619 15:03:00.626623 28134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:00.628356 28119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:03:00.628520 28119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "ade8cd5630504d728cb1bc41c79d0361"
format_stamp: "Formatted at 2026-06-19 15:03:00 on dist-test-slave-zpbw"
server_key: "1ddea50a78c060aed5df770ee868b1c1"
server_key_iv: "7106b87c03cf9aa23f866a5649636c9f"
server_key_version: "encryptionkey@0"
I20260619 15:03:00.628746 28119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:00.644366 28119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:00.648734 28119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:00.649075 28119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:00.650067 28119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:00.651432 28119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:00.651541 28119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:00.651698 28119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:00.651773 28119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:00.686194 28119 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:40091
I20260619 15:03:00.686239 28246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:40091 every 8 connection(s)
I20260619 15:03:00.687969 28119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:03:00.691766 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 28119
I20260619 15:03:00.691969 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:03:00.692713 24989 external_mini_cluster.cc:1428] Setting key 37f48f2052ea4a84fff55d24c2429beb
Jun 19 15:03:00 dist-test-slave-zpbw krb5kdc[28013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881380, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:00.709373 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:00.690590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:46129 (local address 127.24.103.126:38367)
0619 15:03:00.691101 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:00.691106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:00.692221 (+ 1115us) server_negotiation.cc:407] Connection header received
0619 15:03:00.694289 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:00.694294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:00.694386 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:00.694511 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:00.696164 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:00.696899 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:00.698561 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:00.699059 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:00.699204 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:00.703463 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:00.703469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:00.703472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:00.703506 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:00.706069 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:00.706841 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:00.706846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:00.706853 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:00.706970 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:00.707493 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:00.707502 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:00.707506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:00.707810 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:00.707976 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:00.708748 (+ 772us) server_negotiation.cc:299] Negotiation successful
0619 15:03:00.709004 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":297,"threads_started":1}
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260619 15:03:00.710412 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:00.691181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38367 (local address 127.24.103.65:46129)
0619 15:03:00.692020 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:00.692058 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:00.694001 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:00.694766 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:00.694828 (+ 62us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:00.695384 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:00.695980 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:00.695997 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:00.697075 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:00.697080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:00.698399 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:00.698411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:00.699305 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:00.699314 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:00.699569 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:00.700305 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:00.700340 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:00.703225 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:00.706223 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:00.706233 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:00.706250 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:00.706683 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:00.707101 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:00.707106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:00.707110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:00.707375 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:00.707937 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:00.707946 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:00.708308 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 15:03:00.708641 (+ 333us) client_negotiation.cc:240] Negotiation successful
0619 15:03:00.709229 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":376,"threads_started":1}
I20260619 15:03:00.712168 28247 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38367
I20260619 15:03:00.712497 28247 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
I20260619 15:03:00.713289 28247 heartbeater.cc:511] Master 127.24.103.126:38367 requested a full tablet report, sending...
I20260619 15:03:00.715456 28058 ts_manager.cc:194] Registered new tserver with Master: ade8cd5630504d728cb1bc41c79d0361 (127.24.103.65:40091)
I20260619 15:03:00.716908 28058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:46129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:03:00.735576 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38367
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:01.070255 28253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:01.070694 28253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:01.070871 28253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:01.081805 28253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:01.081925 28253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:01.082038 28253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:03:01.091885 28253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:01.093132 28253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:01.094393 28253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:01.108433 28258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:01.108834 28253 server_base.cc:1034] running on GCE node
W20260619 15:03:01.108490 28259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:01.108373 28261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:01.109733 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:01.110829 28253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:01.112069 28253 hybrid_clock.cc:648] HybridClock initialized: now 1781881381112028 us; error 48 us; skew 500 ppm
I20260619 15:03:01.112417 28253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:01 dist-test-slave-zpbw krb5kdc[28013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881381, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:01.115734 28253 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:03:01.117216 28253 webserver.cc:460] Webserver started at http://127.24.103.66:41293/ using document root <none> and password file <none>
I20260619 15:03:01.118003 28253 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:01.118170 28253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:01.118487 28253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:01.121919 28253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "e57329596b624082861131d04724dfc1"
format_stamp: "Formatted at 2026-06-19 15:03:01 on dist-test-slave-zpbw"
server_key: "b221d85f2d0a8f7b7d88422c2027463e"
server_key_iv: "ed75706afe32f5be5130e0e64892b998"
server_key_version: "encryptionkey@0"
I20260619 15:03:01.122749 28253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "e57329596b624082861131d04724dfc1"
format_stamp: "Formatted at 2026-06-19 15:03:01 on dist-test-slave-zpbw"
server_key: "b221d85f2d0a8f7b7d88422c2027463e"
server_key_iv: "ed75706afe32f5be5130e0e64892b998"
server_key_version: "encryptionkey@0"
I20260619 15:03:01.128320 28253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 15:03:01.132140 28268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:01.133702 28253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:03:01.133900 28253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "e57329596b624082861131d04724dfc1"
format_stamp: "Formatted at 2026-06-19 15:03:01 on dist-test-slave-zpbw"
server_key: "b221d85f2d0a8f7b7d88422c2027463e"
server_key_iv: "ed75706afe32f5be5130e0e64892b998"
server_key_version: "encryptionkey@0"
I20260619 15:03:01.134153 28253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:01.160774 28253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:01.165575 28253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:01.165925 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:01.166992 28253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:01.168722 28253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:01.168845 28253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:01.169005 28253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:01.169086 28253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:01.202816 28253 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:44077
I20260619 15:03:01.202884 28380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:44077 every 8 connection(s)
I20260619 15:03:01.204175 28253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:03:01.205210 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 28253
I20260619 15:03:01.205395 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:03:01.205904 24989 external_mini_cluster.cc:1428] Setting key 980bf2750720a55157a268060a0d6c14
Jun 19 15:03:01 dist-test-slave-zpbw krb5kdc[28013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881381, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260619 15:03:01.228413 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:01.206911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:57899 (local address 127.24.103.126:38367)
0619 15:03:01.207122 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:01.207129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:01.208632 (+ 1503us) server_negotiation.cc:407] Connection header received
0619 15:03:01.211583 (+ 2951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:01.211588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:01.211658 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:01.211746 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:01.214013 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.215115 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.216725 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.217494 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.217651 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:01.221312 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:01.221322 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:01.221330 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:01.221378 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:01.225089 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:01.225918 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:01.225924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:01.225928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:01.226083 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:01.226557 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:01.226565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:01.226571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:01.226957 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:01.227075 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:01.227888 (+ 813us) server_negotiation.cc:299] Negotiation successful
0619 15:03:01.228068 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
I20260619 15:03:01.229601 28383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:01.207517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38367 (local address 127.24.103.66:57899)
0619 15:03:01.208434 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:01.208490 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:01.211268 (+ 2778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:01.212060 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:01.212074 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:01.212906 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:01.213812 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:01.213840 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.215290 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.215294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:01.216567 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:01.216584 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.217729 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.217733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:01.217936 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:01.218683 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:01.218707 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:01.221044 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:01.225280 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:01.225288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:01.225303 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:01.225773 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:01.226198 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:01.226201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:01.226203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:01.226392 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:01.227075 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:01.227081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:01.227510 (+ 429us) client_negotiation.cc:769] Sending connection context
0619 15:03:01.227782 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 15:03:01.228255 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":395,"threads_started":1}
I20260619 15:03:01.231254 28381 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38367
I20260619 15:03:01.231573 28381 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:01.232331 28381 heartbeater.cc:511] Master 127.24.103.126:38367 requested a full tablet report, sending...
I20260619 15:03:01.233989 28058 ts_manager.cc:194] Registered new tserver with Master: e57329596b624082861131d04724dfc1 (127.24.103.66:44077)
I20260619 15:03:01.235283 28058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:57899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:03:01.254616 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38367
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:01.585984 28386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:01.586287 28386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:01.586381 28386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:01.593734 28386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:01.593858 28386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:01.593972 28386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:03:01.604131 28386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:01.605410 28386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:01.606623 28386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:01.619941 28391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:01.619966 28392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:01.619997 28394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:01.626135 28386 server_base.cc:1034] running on GCE node
I20260619 15:03:01.626873 28386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:01.627843 28386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:01.629067 28386 hybrid_clock.cc:648] HybridClock initialized: now 1781881381629015 us; error 54 us; skew 500 ppm
I20260619 15:03:01.629400 28386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:01 dist-test-slave-zpbw krb5kdc[28013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881381, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:01.632875 28386 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:03:01.634456 28386 webserver.cc:460] Webserver started at http://127.24.103.67:40513/ using document root <none> and password file <none>
I20260619 15:03:01.635391 28386 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:01.635598 28386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:01.635993 28386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:01.639037 28386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "b16353260a4c4860ae096ea4155e7cc5"
format_stamp: "Formatted at 2026-06-19 15:03:01 on dist-test-slave-zpbw"
server_key: "1f736d046b5d4bbe949d8450bdfd2866"
server_key_iv: "10399ea98a34aea0250636e65fe94d1d"
server_key_version: "encryptionkey@0"
I20260619 15:03:01.639859 28386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "b16353260a4c4860ae096ea4155e7cc5"
format_stamp: "Formatted at 2026-06-19 15:03:01 on dist-test-slave-zpbw"
server_key: "1f736d046b5d4bbe949d8450bdfd2866"
server_key_iv: "10399ea98a34aea0250636e65fe94d1d"
server_key_version: "encryptionkey@0"
I20260619 15:03:01.645236 28386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:03:01.648882 28401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:01.650313 28386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:03:01.650465 28386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "b16353260a4c4860ae096ea4155e7cc5"
format_stamp: "Formatted at 2026-06-19 15:03:01 on dist-test-slave-zpbw"
server_key: "1f736d046b5d4bbe949d8450bdfd2866"
server_key_iv: "10399ea98a34aea0250636e65fe94d1d"
server_key_version: "encryptionkey@0"
I20260619 15:03:01.650642 28386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:01.671324 28386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:01.675853 28386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:01.676205 28386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:01.677436 28386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:01.679067 28386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:01.679172 28386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:01.679299 28386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:01.679378 28386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:01.713888 28386 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:46701
I20260619 15:03:01.713943 28513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:46701 every 8 connection(s)
I20260619 15:03:01.715344 28386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:03:01.720222 28247 heartbeater.cc:503] Master 127.24.103.126:38367 was elected leader, sending a full tablet report...
I20260619 15:03:01.724915 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 28386
I20260619 15:03:01.725100 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:03:01.725678 24989 external_mini_cluster.cc:1428] Setting key 3559472e41776194beb7ae7a97d7024c
Jun 19 15:03:01 dist-test-slave-zpbw krb5kdc[28013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881381, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:01.737021 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:01.717981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:42703 (local address 127.24.103.126:38367)
0619 15:03:01.718196 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:01.718202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:01.719463 (+ 1261us) server_negotiation.cc:407] Connection header received
0619 15:03:01.721670 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:01.721675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:01.721751 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:01.721869 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:01.723540 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.724275 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.726699 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.727236 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.727331 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:01.731211 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:01.731216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:01.731219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:01.731251 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:01.733896 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:01.734631 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:01.734637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:01.734640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:01.734768 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:01.735259 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:01.735265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:01.735268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:01.735549 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:01.735710 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:01.736439 (+ 729us) server_negotiation.cc:299] Negotiation successful
0619 15:03:01.736670 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 15:03:01.737988 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:01.718523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38367 (local address 127.24.103.67:42703)
0619 15:03:01.719278 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:01.719316 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:01.721401 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:01.722176 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:01.722190 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:01.722742 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:01.723388 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:01.723405 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.724461 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.724467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:01.726523 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:01.726537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.727468 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.727476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:01.727796 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:01.728590 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:01.728629 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:01.730986 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:01.734049 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:01.734058 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:01.734076 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:01.734494 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:01.734932 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:01.734937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:01.734941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:01.735128 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:01.735699 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:01.735706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:01.736057 (+ 351us) client_negotiation.cc:769] Sending connection context
0619 15:03:01.736338 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 15:03:01.736897 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":332,"threads_started":1}
I20260619 15:03:01.739790 28514 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38367
I20260619 15:03:01.740165 28514 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:01.741061 28514 heartbeater.cc:511] Master 127.24.103.126:38367 requested a full tablet report, sending...
I20260619 15:03:01.742560 28058 ts_manager.cc:194] Registered new tserver with Master: b16353260a4c4860ae096ea4155e7cc5 (127.24.103.67:46701)
I20260619 15:03:01.743453 28058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:42703
I20260619 15:03:01.753965 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 15:03:01 dist-test-slave-zpbw krb5kdc[28013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881381, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 15:03:01 dist-test-slave-zpbw krb5kdc[28013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881381, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:01.802877 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:01.789107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.24.103.126:38367)
0619 15:03:01.789281 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:01.789285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:01.789714 (+ 429us) server_negotiation.cc:407] Connection header received
0619 15:03:01.789877 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:01.789881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:01.789944 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:01.790034 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:01.791431 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.792137 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.793454 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.793870 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.793956 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:01.797807 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:01.797813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:01.797816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:01.797847 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:01.800518 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:01.801107 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:01.801111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:01.801113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:01.801262 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:01.801642 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:01.801645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:01.801647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:01.801940 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:01.802206 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:01.802432 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 15:03:01.802586 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 15:03:01.803620 28058 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56236
I20260619 15:03:01.805214 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 28119
Jun 19 15:03:01 dist-test-slave-zpbw krb5kdc[28013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881381, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:03:01.822202 28525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:01.806882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.24.103.65:40091)
0619 15:03:01.807491 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:01.807498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:01.807523 (+ 25us) server_negotiation.cc:407] Connection header received
0619 15:03:01.807578 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:01.807584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:01.807814 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:01.807936 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:01.809213 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.810210 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.811421 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.811969 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.812065 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:01.816309 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:01.816321 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:01.816334 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:01.816365 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:01.819302 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:01.820039 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:01.820046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:01.820049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:01.820195 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:01.820611 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:01.820616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:01.820623 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:01.820965 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:01.821286 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:01.821581 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 15:03:01.821845 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":306,"threads_started":1}
W20260619 15:03:01.823510 28242 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60412
W20260619 15:03:01.824600 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 28119: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:03:01.824783 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 28119
I20260619 15:03:01.840689 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 28253
Jun 19 15:03:01 dist-test-slave-zpbw krb5kdc[28013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881381, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:03:01.857550 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:01.841605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.24.103.66:44077)
0619 15:03:01.842220 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:01.842227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:01.842264 (+ 37us) server_negotiation.cc:407] Connection header received
0619 15:03:01.842340 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:01.842346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:01.842557 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:01.842691 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:01.843970 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.845009 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.846223 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.847006 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.847116 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:01.851416 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:01.851428 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:01.851444 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:01.851480 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:01.854525 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:01.855339 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:01.855347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:01.855349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:01.855476 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:01.855880 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:01.855889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:01.855892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:01.856247 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:01.856575 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:01.856899 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 15:03:01.857177 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":327,"threads_started":1}
W20260619 15:03:01.859179 28376 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43938
W20260619 15:03:01.860342 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 28253: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:03:01.860458 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 28253
I20260619 15:03:01.877475 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 28386
Jun 19 15:03:01 dist-test-slave-zpbw krb5kdc[28013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881381, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:03:01.893709 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:01.878546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.24.103.67:46701)
0619 15:03:01.879249 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:01.879256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:01.879285 (+ 29us) server_negotiation.cc:407] Connection header received
0619 15:03:01.879349 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:01.879355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:01.879627 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:01.879770 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:01.881098 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.882159 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.883512 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:01.884089 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:01.884192 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:01.888582 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:01.888596 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:01.888611 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:01.888652 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:01.891327 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:01.891965 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:01.891971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:01.891973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:01.892099 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:01.892395 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:01.892398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:01.892400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:01.892726 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:01.892952 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:01.893227 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 15:03:01.893422 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":408,"threads_started":1}
W20260619 15:03:01.895080 28509 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48266
W20260619 15:03:01.896234 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 28386: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:03:01.896450 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 28386
I20260619 15:03:01.911573 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 28026
I20260619 15:03:01.912766 28088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:02.016415 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 28026
2026-06-19T15:03:02Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3711 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:03:02 dist-test-slave-zpbw krb5kdc[28531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:03:02 dist-test-slave-zpbw krb5kdc[28531](info): set up 2 sockets
Jun 19 15:03:02 dist-test-slave-zpbw krb5kdc[28531](info): commencing operation
krb5kdc: starting...
W20260619 15:03:03.099391 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:03:03 dist-test-slave-zpbw krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:03:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:03:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:03:03.302269 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35785
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:35785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:03.630312 28544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:03.630632 28544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:03.630721 28544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:03.638209 28544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:03.638343 28544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:03.638391 28544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:03.638442 28544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:03.638487 28544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:03:03.648993 28544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:35785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:03.650295 28544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:03.651688 28544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:03.664602 28549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:03.664601 28552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:03.664661 28550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:03.668386 28544 server_base.cc:1034] running on GCE node
I20260619 15:03:03.669305 28544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:03.670925 28544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:03.672195 28544 hybrid_clock.cc:648] HybridClock initialized: now 1781881383672147 us; error 56 us; skew 500 ppm
I20260619 15:03:03.672632 28544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:03 dist-test-slave-zpbw krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881383, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:03.676462 28544 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:03:03.677992 28544 webserver.cc:460] Webserver started at http://127.24.103.126:34667/ using document root <none> and password file <none>
I20260619 15:03:03.678710 28544 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:03.678880 28544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:03.679170 28544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:03.682283 28544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "240e339a77784dd7859d283a76ff532a"
format_stamp: "Formatted at 2026-06-19 15:03:03 on dist-test-slave-zpbw"
server_key: "d5420a15d992c7c8fc6d207f6702ed98"
server_key_iv: "fbd0885b63790c5ee06d5d73890c81c1"
server_key_version: "encryptionkey@0"
I20260619 15:03:03.683444 28544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "240e339a77784dd7859d283a76ff532a"
format_stamp: "Formatted at 2026-06-19 15:03:03 on dist-test-slave-zpbw"
server_key: "d5420a15d992c7c8fc6d207f6702ed98"
server_key_iv: "fbd0885b63790c5ee06d5d73890c81c1"
server_key_version: "encryptionkey@0"
I20260619 15:03:03.689653 28544 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 15:03:03.693302 28559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:03.694670 28544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 15:03:03.694886 28544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "240e339a77784dd7859d283a76ff532a"
format_stamp: "Formatted at 2026-06-19 15:03:03 on dist-test-slave-zpbw"
server_key: "d5420a15d992c7c8fc6d207f6702ed98"
server_key_iv: "fbd0885b63790c5ee06d5d73890c81c1"
server_key_version: "encryptionkey@0"
I20260619 15:03:03.695106 28544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:03.717886 28544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:03.722954 28544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:03.723263 28544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:03.742821 28544 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:35785
I20260619 15:03:03.742892 28610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:35785 every 8 connection(s)
I20260619 15:03:03.744248 28544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:03.748579 28611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:03.752600 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 28544
I20260619 15:03:03.752852 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:03:03.753504 24989 external_mini_cluster.cc:1428] Setting key ff68203ff3b8ede2d6470a554d28c7b2
I20260619 15:03:03.764104 28611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a: Bootstrap starting.
I20260619 15:03:03.768177 28611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:03.769342 28611 log.cc:824] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a: Log is configured to *not* fsync() on all Append() calls
Jun 19 15:03:03 dist-test-slave-zpbw krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:03.772540 28611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a: No bootstrap required, opened a new log
I20260619 15:03:03.779392 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:03.755762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.24.103.126:35785)
0619 15:03:03.756687 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:03.756705 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:03.756759 (+ 54us) server_negotiation.cc:407] Connection header received
0619 15:03:03.759101 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:03.759141 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:03.759819 (+ 678us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:03.760347 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:03.761743 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:03.763971 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:03.765521 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:03.766806 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:03.767009 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:03.771349 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:03.771358 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:03.771375 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:03.771416 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:03.774956 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:03.775753 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:03.775769 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:03.775776 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:03.775947 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:03.776358 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:03.776364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:03.776367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:03.776971 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:03.777316 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:03.777773 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 15:03:03.778225 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":366,"threads_started":1}
I20260619 15:03:03.781141 28611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240e339a77784dd7859d283a76ff532a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35785 } }
I20260619 15:03:03.781543 28611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:03.781672 28611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240e339a77784dd7859d283a76ff532a, State: Initialized, Role: FOLLOWER
I20260619 15:03:03.782356 28611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240e339a77784dd7859d283a76ff532a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35785 } }
I20260619 15:03:03.782630 28611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:03.782819 28611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:03.782970 28611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:03.785076 28611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240e339a77784dd7859d283a76ff532a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35785 } }
I20260619 15:03:03.785674 28611 leader_election.cc:302] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 240e339a77784dd7859d283a76ff532a; no voters:
I20260619 15:03:03.786392 28611 leader_election.cc:288] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:03:03.786600 28616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:03.787295 28616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [term 1 LEADER]: Becoming Leader. State: Replica: 240e339a77784dd7859d283a76ff532a, State: Running, Role: LEADER
I20260619 15:03:03.787876 28616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240e339a77784dd7859d283a76ff532a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35785 } }
I20260619 15:03:03.788589 28611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:03.790961 28618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 240e339a77784dd7859d283a76ff532a. Latest consensus state: current_term: 1 leader_uuid: "240e339a77784dd7859d283a76ff532a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240e339a77784dd7859d283a76ff532a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35785 } } }
I20260619 15:03:03.790935 28617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "240e339a77784dd7859d283a76ff532a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240e339a77784dd7859d283a76ff532a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35785 } } }
I20260619 15:03:03.791414 28618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:03.791442 28617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:03.795965 28622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:03.801669 28622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:03.811045 28622 catalog_manager.cc:1347] Generated new cluster ID: c7fb89c9e7144f18ad0c6b55ce30945e
I20260619 15:03:03.811175 28622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:03.820657 28622 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:03:03.822016 28622 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:03.829741 28622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 240e339a77784dd7859d283a76ff532a: Generated new TSK 0
I20260619 15:03:03.830865 28622 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:03:03.882334 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35785
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:04.223688 28637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:04.223979 28637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:04.224144 28637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:04.232115 28637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:04.232231 28637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:04.232451 28637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:03:04.243160 28637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:04.244621 28637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:04.246084 28637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:04.261037 28645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:04.261276 28643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:04.261245 28642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:04.261526 28637 server_base.cc:1034] running on GCE node
I20260619 15:03:04.262511 28637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:04.263598 28637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:04.264840 28637 hybrid_clock.cc:648] HybridClock initialized: now 1781881384264782 us; error 51 us; skew 500 ppm
I20260619 15:03:04.265178 28637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:04 dist-test-slave-zpbw krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881384, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:04.268414 28637 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:03:04.269904 28637 webserver.cc:460] Webserver started at http://127.24.103.65:40965/ using document root <none> and password file <none>
I20260619 15:03:04.270632 28637 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:04.270780 28637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:04.271190 28637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:04.274627 28637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "1536476f171046c4ab543c6317a87b48"
format_stamp: "Formatted at 2026-06-19 15:03:04 on dist-test-slave-zpbw"
server_key: "4ac015e4223f8dae39c8435c0a798d4b"
server_key_iv: "be4f90edb39d619c0bcefb56a21c11b2"
server_key_version: "encryptionkey@0"
I20260619 15:03:04.275496 28637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "1536476f171046c4ab543c6317a87b48"
format_stamp: "Formatted at 2026-06-19 15:03:04 on dist-test-slave-zpbw"
server_key: "4ac015e4223f8dae39c8435c0a798d4b"
server_key_iv: "be4f90edb39d619c0bcefb56a21c11b2"
server_key_version: "encryptionkey@0"
I20260619 15:03:04.281071 28637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:03:04.284701 28652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:04.286130 28637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:03:04.286339 28637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "1536476f171046c4ab543c6317a87b48"
format_stamp: "Formatted at 2026-06-19 15:03:04 on dist-test-slave-zpbw"
server_key: "4ac015e4223f8dae39c8435c0a798d4b"
server_key_iv: "be4f90edb39d619c0bcefb56a21c11b2"
server_key_version: "encryptionkey@0"
I20260619 15:03:04.286553 28637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:04.303923 28637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:04.308743 28637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:04.309053 28637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:04.310096 28637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:04.311507 28637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:04.311582 28637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:04.311729 28637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:04.311811 28637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:04.346064 28637 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:41587
I20260619 15:03:04.346100 28764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:41587 every 8 connection(s)
I20260619 15:03:04.347502 28637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:03:04.353099 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 28637
I20260619 15:03:04.353328 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:03:04.353936 24989 external_mini_cluster.cc:1428] Setting key 60ea3fce0815a78413e269762053a761
Jun 19 15:03:04 dist-test-slave-zpbw krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881384, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:04.369153 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:04.350302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:45815 (local address 127.24.103.126:35785)
0619 15:03:04.350743 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:04.350748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:04.351878 (+ 1130us) server_negotiation.cc:407] Connection header received
0619 15:03:04.353936 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:04.353942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:04.354085 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:04.354219 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:04.356117 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:04.356843 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:04.358483 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:04.358975 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:04.359123 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:04.363121 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:04.363126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:04.363143 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:04.363178 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:04.365871 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:04.366695 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:04.366706 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:04.366709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:04.366891 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:04.367353 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:04.367358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:04.367361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:04.367650 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:04.367818 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:04.368575 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 15:03:04.368801 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":232,"threads_started":1}
I20260619 15:03:04.370210 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:04.350944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35785 (local address 127.24.103.65:45815)
0619 15:03:04.351706 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:04.351747 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:04.353634 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:04.354518 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:04.354531 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:04.355187 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:04.355920 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:04.355946 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:04.357010 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:04.357018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:04.358317 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:04.358330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:04.359230 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:04.359238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:04.359503 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:04.360251 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:04.360274 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:04.362840 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:04.366068 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:04.366078 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:04.366098 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:04.366552 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:04.367026 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:04.367031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:04.367035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:04.367229 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:04.367803 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:04.367812 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:04.368170 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 15:03:04.368458 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 15:03:04.368998 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":270,"threads_started":1}
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260619 15:03:04.371868 28765 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35785
I20260619 15:03:04.372206 28765 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:04.372951 28765 heartbeater.cc:511] Master 127.24.103.126:35785 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
I20260619 15:03:04.375311 28576 ts_manager.cc:194] Registered new tserver with Master: 1536476f171046c4ab543c6317a87b48 (127.24.103.65:41587)
I20260619 15:03:04.376793 28576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:45815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:03:04.396533 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35785
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:04.728227 28771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:04.728521 28771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:04.728626 28771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:04.737123 28771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:04.737252 28771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:04.737368 28771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:03:04.747732 28771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:04.749071 28771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:04.750545 28771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:04.765128 28779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:04.765074 28777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:04.765202 28776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:04.766074 28771 server_base.cc:1034] running on GCE node
I20260619 15:03:04.766710 28771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:04.767763 28771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:04.768970 28771 hybrid_clock.cc:648] HybridClock initialized: now 1781881384768909 us; error 60 us; skew 500 ppm
I20260619 15:03:04.769258 28771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:04 dist-test-slave-zpbw krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881384, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:04.779013 28771 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:03:04.780774 28771 webserver.cc:460] Webserver started at http://127.24.103.66:39235/ using document root <none> and password file <none>
I20260619 15:03:04.781560 28771 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:04.781735 28771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:04.782069 28771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:04.785347 28771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "de5104f608e243ab840a198da30f2df5"
format_stamp: "Formatted at 2026-06-19 15:03:04 on dist-test-slave-zpbw"
server_key: "dc533991851fb71066867b63b6981e29"
server_key_iv: "33c3cfc0797de6d98a154126ca8f449b"
server_key_version: "encryptionkey@0"
I20260619 15:03:04.786218 28771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "de5104f608e243ab840a198da30f2df5"
format_stamp: "Formatted at 2026-06-19 15:03:04 on dist-test-slave-zpbw"
server_key: "dc533991851fb71066867b63b6981e29"
server_key_iv: "33c3cfc0797de6d98a154126ca8f449b"
server_key_version: "encryptionkey@0"
I20260619 15:03:04.791824 28771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:03:04.795450 28786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:04.797147 28771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:03:04.797319 28771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "de5104f608e243ab840a198da30f2df5"
format_stamp: "Formatted at 2026-06-19 15:03:04 on dist-test-slave-zpbw"
server_key: "dc533991851fb71066867b63b6981e29"
server_key_iv: "33c3cfc0797de6d98a154126ca8f449b"
server_key_version: "encryptionkey@0"
I20260619 15:03:04.797515 28771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:04.816538 28771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:04.821115 28771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:04.821444 28771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:04.822453 28771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:04.823935 28771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:04.824007 28771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:04.824165 28771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:04.824215 28771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:04.857823 28771 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:46297
I20260619 15:03:04.857856 28898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:46297 every 8 connection(s)
I20260619 15:03:04.859234 28771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:03:04.867596 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 28771
I20260619 15:03:04.867805 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:03:04.868330 24989 external_mini_cluster.cc:1428] Setting key f67913bbaf359d3a4cac51499cb23403
Jun 19 15:03:04 dist-test-slave-zpbw krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881384, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:04.884706 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:04.861792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:41487 (local address 127.24.103.126:35785)
0619 15:03:04.862002 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:04.862009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:04.863365 (+ 1356us) server_negotiation.cc:407] Connection header received
0619 15:03:04.865150 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:04.865155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:04.865231 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:04.865325 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:04.866944 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:04.868004 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:04.869840 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:04.870502 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:04.870654 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:04.875505 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:04.875511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:04.875514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:04.875552 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:04.879240 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:04.880768 (+ 1528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:04.880778 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:04.880785 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:04.880979 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:04.881646 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:04.881654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:04.881660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:04.882116 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:04.882317 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:04.883517 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0619 15:03:04.884015 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 15:03:04.885802 28901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:04.862418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35785 (local address 127.24.103.66:41487)
0619 15:03:04.863186 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:04.863234 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:04.864879 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:04.865550 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:04.865558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:04.866098 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:04.866690 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:04.866708 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:04.868157 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:04.868161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:04.869697 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:04.869709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:04.870803 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:04.870812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:04.871126 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:04.871885 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:04.871927 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:04.875189 (+ 3262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:04.879673 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:04.879687 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:04.879710 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:04.880523 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:04.881139 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:04.881147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:04.881169 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:04.881464 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:04.882288 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:04.882300 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:04.882913 (+ 613us) client_negotiation.cc:769] Sending connection context
0619 15:03:04.883247 (+ 334us) client_negotiation.cc:240] Negotiation successful
0619 15:03:04.883876 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":323,"threads_started":1}
I20260619 15:03:04.888051 28899 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35785
I20260619 15:03:04.888499 28899 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:04.889582 28899 heartbeater.cc:511] Master 127.24.103.126:35785 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260619 15:03:04.891541 28576 ts_manager.cc:194] Registered new tserver with Master: de5104f608e243ab840a198da30f2df5 (127.24.103.66:46297)
I20260619 15:03:04.892405 28576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:41487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:03:04.917397 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35785
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:05.252748 28904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:05.253052 28904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:05.253149 28904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:05.261055 28904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:05.261183 28904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:05.261301 28904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:03:05.271468 28904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:05.272782 28904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:05.274072 28904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:05.289309 28909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:05.289218 28912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:05.289338 28910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:05.290678 28904 server_base.cc:1034] running on GCE node
I20260619 15:03:05.291472 28904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:05.292547 28904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:05.293825 28904 hybrid_clock.cc:648] HybridClock initialized: now 1781881385293753 us; error 52 us; skew 500 ppm
I20260619 15:03:05.294196 28904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:05 dist-test-slave-zpbw krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881385, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:05.297811 28904 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:03:05.299656 28904 webserver.cc:460] Webserver started at http://127.24.103.67:38907/ using document root <none> and password file <none>
I20260619 15:03:05.300395 28904 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:05.300550 28904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:05.300864 28904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:05.304116 28904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "4f7ee8b6aa5f42e1a9355b3a66a52c1a"
format_stamp: "Formatted at 2026-06-19 15:03:05 on dist-test-slave-zpbw"
server_key: "be0354901bf3ef42e48d759019752698"
server_key_iv: "f65e91f0696bed6cc5f9bc70949c8737"
server_key_version: "encryptionkey@0"
I20260619 15:03:05.304982 28904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "4f7ee8b6aa5f42e1a9355b3a66a52c1a"
format_stamp: "Formatted at 2026-06-19 15:03:05 on dist-test-slave-zpbw"
server_key: "be0354901bf3ef42e48d759019752698"
server_key_iv: "f65e91f0696bed6cc5f9bc70949c8737"
server_key_version: "encryptionkey@0"
I20260619 15:03:05.310659 28904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 15:03:05.314286 28919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:05.315899 28904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:03:05.316155 28904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "4f7ee8b6aa5f42e1a9355b3a66a52c1a"
format_stamp: "Formatted at 2026-06-19 15:03:05 on dist-test-slave-zpbw"
server_key: "be0354901bf3ef42e48d759019752698"
server_key_iv: "f65e91f0696bed6cc5f9bc70949c8737"
server_key_version: "encryptionkey@0"
I20260619 15:03:05.316435 28904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:05.334175 28904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:05.339890 28904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:05.340269 28904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:05.341405 28904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:05.343128 28904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:05.343266 28904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:05.343466 28904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:05.343560 28904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:05.379930 28765 heartbeater.cc:503] Master 127.24.103.126:35785 was elected leader, sending a full tablet report...
I20260619 15:03:05.382483 28904 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:37541
I20260619 15:03:05.382504 29031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:37541 every 8 connection(s)
I20260619 15:03:05.384079 28904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:03:05.388250 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 28904
I20260619 15:03:05.388465 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:03:05.389022 24989 external_mini_cluster.cc:1428] Setting key 94297eba31d9c568cea75fba335f0cb2
Jun 19 15:03:05 dist-test-slave-zpbw krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881385, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:05.407178 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:05.386746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:48813 (local address 127.24.103.126:35785)
0619 15:03:05.387274 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:05.387279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:05.388305 (+ 1026us) server_negotiation.cc:407] Connection header received
0619 15:03:05.390906 (+ 2601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:05.390911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:05.391008 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:05.391161 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:05.393298 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.394203 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.396181 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.396673 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.396842 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:05.400907 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:05.400912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:05.400916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:05.400951 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:05.403862 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:05.404631 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:05.404638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:05.404642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:05.404770 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:05.405221 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:05.405226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:05.405229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:05.405536 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:05.405694 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:05.406422 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 15:03:05.406767 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":281,"threads_started":1}
I20260619 15:03:05.408151 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:05.387301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35785 (local address 127.24.103.67:48813)
0619 15:03:05.388099 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:05.388167 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:05.390530 (+ 2363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:05.391581 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:05.391615 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:05.392447 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:05.393094 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:05.393115 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.394396 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.394403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:05.396005 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:05.396028 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.396943 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.396954 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:05.397261 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:05.397971 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:05.398001 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:05.400674 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:05.404021 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:05.404031 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:05.404052 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:05.404495 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:05.404887 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:05.404892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:05.404896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:05.405098 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:05.405672 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:05.405680 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:05.406049 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 15:03:05.406322 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 15:03:05.406995 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":268,"threads_started":1}
I20260619 15:03:05.409709 29032 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35785
I20260619 15:03:05.410001 29032 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:05.410719 29032 heartbeater.cc:511] Master 127.24.103.126:35785 requested a full tablet report, sending...
I20260619 15:03:05.412231 28576 ts_manager.cc:194] Registered new tserver with Master: 4f7ee8b6aa5f42e1a9355b3a66a52c1a (127.24.103.67:37541)
I20260619 15:03:05.413064 28576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:48813
I20260619 15:03:05.417470 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 15:03:05 dist-test-slave-zpbw krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 15:03:05 dist-test-slave-zpbw krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:03:05.467744 29040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:05.452145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.24.103.65:41587)
0619 15:03:05.452889 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:05.452897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:05.452922 (+ 25us) server_negotiation.cc:407] Connection header received
0619 15:03:05.452982 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:05.452988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:05.453223 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:05.453353 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:05.454742 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.455976 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.457213 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.457881 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.457978 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:05.461933 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:05.461947 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:05.461962 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:05.462000 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:05.464782 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:05.465546 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:05.465554 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:05.465556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:05.465686 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:05.466080 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:05.466087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:05.466090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:05.466451 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:05.466763 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:05.467088 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 15:03:05.467373 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":413,"threads_started":1}
Jun 19 15:03:05 dist-test-slave-zpbw krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:05.486032 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:05.472107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.24.103.126:35785)
0619 15:03:05.472338 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:05.472344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:05.472473 (+ 129us) server_negotiation.cc:407] Connection header received
0619 15:03:05.472681 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:05.472691 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:05.472757 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:05.472864 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:05.474135 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.474915 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.476180 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.476608 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.476746 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:05.480793 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:05.480799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:05.480802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:05.480836 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:05.483410 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:05.484093 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:05.484099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:05.484107 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:05.484231 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:05.484604 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:05.484610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:05.484613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:05.484944 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:05.485208 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:05.485497 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 15:03:05.485721 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260619 15:03:05.486687 28576 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46106
I20260619 15:03:05.487748 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 28637
I20260619 15:03:05.500923 29040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:05.488597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.24.103.65:41587)
0619 15:03:05.488827 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:05.488834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:05.489040 (+ 206us) server_negotiation.cc:407] Connection header received
0619 15:03:05.489166 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:05.489170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:05.489229 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:05.489305 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:05.490550 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.491338 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.492599 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.493007 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.493100 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:05.495522 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:05.495527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:05.495530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:05.495560 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:05.498203 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:05.499111 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:05.499115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:05.499117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:05.499237 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:05.499615 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:05.499619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:05.499621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:05.499933 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:05.500099 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:05.500421 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 15:03:05.500581 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 15:03:05.501498 28760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:05.583703 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 28637
I20260619 15:03:05.599808 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 28771
Jun 19 15:03:05 dist-test-slave-zpbw krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:03:05.615926 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:05.600671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.24.103.66:46297)
0619 15:03:05.601303 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:05.601311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:05.601335 (+ 24us) server_negotiation.cc:407] Connection header received
0619 15:03:05.601399 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:05.601405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:05.601610 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:05.601747 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:05.603166 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.604201 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.605425 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.605975 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.606079 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:05.610150 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:05.610168 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:05.610183 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:05.610220 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:05.612980 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:05.613716 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:05.613724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:05.613727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:05.613850 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:05.614237 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:05.614244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:05.614247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:05.614594 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:05.614961 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:05.615306 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 15:03:05.615573 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":337,"threads_started":1}
I20260619 15:03:05.617211 28894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:05.701700 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 28771
I20260619 15:03:05.717286 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 28904
Jun 19 15:03:05 dist-test-slave-zpbw krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:03:05.734274 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:05.718326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.24.103.67:37541)
0619 15:03:05.718979 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:05.718986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:05.719018 (+ 32us) server_negotiation.cc:407] Connection header received
0619 15:03:05.719081 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:05.719087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:05.719292 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:05.719426 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:05.720778 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.721969 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.723447 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:05.724010 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:05.724111 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:05.728660 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:05.728674 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:05.728689 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:05.728726 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:05.731584 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:05.732264 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:05.732271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:05.732273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:05.732391 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:05.732696 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:05.732699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:05.732701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:05.733036 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:05.733257 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:05.733656 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 15:03:05.733950 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":289,"threads_started":1}
I20260619 15:03:05.735709 29027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:05.820137 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 28904
I20260619 15:03:05.835807 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 28544
I20260619 15:03:05.837090 28606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:05.940903 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 28544
2026-06-19T15:03:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3928 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:03:06 dist-test-slave-zpbw krb5kdc[29052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:03:06 dist-test-slave-zpbw krb5kdc[29052](info): set up 2 sockets
Jun 19 15:03:06 dist-test-slave-zpbw krb5kdc[29052](info): commencing operation
krb5kdc: starting...
W20260619 15:03:07.071553 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:03:07 dist-test-slave-zpbw krb5kdc[29052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:03:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:03:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:03:07.274967 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35255
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:35255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:07.601322 29065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:07.601627 29065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:07.601804 29065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:07.609134 29065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:07.609246 29065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:07.609339 29065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:07.609441 29065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:07.609521 29065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:03:07.619670 29065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:35255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:07.621006 29065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:07.622361 29065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:07.635700 29070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:07.635615 29073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:07.636433 29065 server_base.cc:1034] running on GCE node
W20260619 15:03:07.636654 29071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:07.637840 29065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:07.640066 29065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:07.641326 29065 hybrid_clock.cc:648] HybridClock initialized: now 1781881387641250 us; error 70 us; skew 500 ppm
I20260619 15:03:07.641712 29065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:07 dist-test-slave-zpbw krb5kdc[29052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881387, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:07.645716 29065 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:03:07.647802 29065 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:07.647930 29065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:07.648285 29065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:07.651473 29065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "9ab9270032e842c8869fdf89ba580a7e"
format_stamp: "Formatted at 2026-06-19 15:03:07 on dist-test-slave-zpbw"
server_key: "6497d166432aee9136078878c3176557"
server_key_iv: "005902a391475aef1bf9adba99bf6004"
server_key_version: "encryptionkey@0"
I20260619 15:03:07.652314 29065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "9ab9270032e842c8869fdf89ba580a7e"
format_stamp: "Formatted at 2026-06-19 15:03:07 on dist-test-slave-zpbw"
server_key: "6497d166432aee9136078878c3176557"
server_key_iv: "005902a391475aef1bf9adba99bf6004"
server_key_version: "encryptionkey@0"
I20260619 15:03:07.657831 29065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 15:03:07.661550 29079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:07.663113 29065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:03:07.663309 29065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "9ab9270032e842c8869fdf89ba580a7e"
format_stamp: "Formatted at 2026-06-19 15:03:07 on dist-test-slave-zpbw"
server_key: "6497d166432aee9136078878c3176557"
server_key_iv: "005902a391475aef1bf9adba99bf6004"
server_key_version: "encryptionkey@0"
I20260619 15:03:07.663566 29065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:07.703655 29065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:07.708185 29065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:07.708500 29065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:07.727982 29065 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:35255
I20260619 15:03:07.728001 29130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:35255 every 8 connection(s)
I20260619 15:03:07.728967 29065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:07.732853 29131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:07.736578 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 29065
I20260619 15:03:07.736819 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:03:07.737488 24989 external_mini_cluster.cc:1428] Setting key 4ebdfb4c6900c4bb1c2da252e93d4f7d
I20260619 15:03:07.744468 29131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e: Bootstrap starting.
I20260619 15:03:07.747834 29131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:07.748921 29131 log.cc:824] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:07.752106 29131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e: No bootstrap required, opened a new log
Jun 19 15:03:07 dist-test-slave-zpbw krb5kdc[29052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:07.760896 29131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab9270032e842c8869fdf89ba580a7e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35255 } }
I20260619 15:03:07.761453 29131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:07.761590 29131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ab9270032e842c8869fdf89ba580a7e, State: Initialized, Role: FOLLOWER
I20260619 15:03:07.762283 29131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab9270032e842c8869fdf89ba580a7e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35255 } }
I20260619 15:03:07.762590 29131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:07.762802 29131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:07.762812 29134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:07.739654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.24.103.126:35255)
0619 15:03:07.740417 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:07.740429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:07.740497 (+ 68us) server_negotiation.cc:407] Connection header received
0619 15:03:07.743087 (+ 2590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:07.743113 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:07.743561 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:07.743969 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:07.745682 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:07.747245 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:07.748616 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:07.749425 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:07.749654 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:07.753932 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:07.753944 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:07.753963 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:07.754005 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:07.757341 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:07.758219 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:07.758230 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:07.758237 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:07.758500 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:07.758969 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:07.758975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:07.758991 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:07.759847 (+ 856us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:07.760197 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:07.760720 (+ 523us) server_negotiation.cc:299] Negotiation successful
0619 15:03:07.761207 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":354,"threads_started":1}
I20260619 15:03:07.763056 29131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:07.766011 29131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab9270032e842c8869fdf89ba580a7e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35255 } }
I20260619 15:03:07.766664 29131 leader_election.cc:302] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ab9270032e842c8869fdf89ba580a7e; no voters:
I20260619 15:03:07.767344 29131 leader_election.cc:288] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:03:07.767561 29136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:07.768388 29136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [term 1 LEADER]: Becoming Leader. State: Replica: 9ab9270032e842c8869fdf89ba580a7e, State: Running, Role: LEADER
I20260619 15:03:07.768913 29136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab9270032e842c8869fdf89ba580a7e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35255 } }
I20260619 15:03:07.769631 29131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:07.771971 29138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ab9270032e842c8869fdf89ba580a7e. Latest consensus state: current_term: 1 leader_uuid: "9ab9270032e842c8869fdf89ba580a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab9270032e842c8869fdf89ba580a7e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35255 } } }
I20260619 15:03:07.772398 29138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:07.772608 29137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ab9270032e842c8869fdf89ba580a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab9270032e842c8869fdf89ba580a7e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35255 } } }
I20260619 15:03:07.773309 29137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:07.775636 29141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:07.782809 29141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:07.792959 29141 catalog_manager.cc:1347] Generated new cluster ID: 1e4fad6751ad403a82274d33b53aeb3e
I20260619 15:03:07.793089 29141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:07.814426 29141 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:03:07.816414 29141 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:07.832829 29141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e: Generated new TSK 0
I20260619 15:03:07.833982 29141 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:03:07.898428 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35255
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:08.241470 29157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:08.241776 29157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:08.241942 29157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:08.249562 29157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:08.249693 29157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:08.249882 29157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:03:08.260267 29157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:08.261648 29157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:08.263057 29157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:08.277523 29163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:08.277666 29165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:08.277549 29162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:08.278293 29157 server_base.cc:1034] running on GCE node
I20260619 15:03:08.279024 29157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:08.280031 29157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:08.281239 29157 hybrid_clock.cc:648] HybridClock initialized: now 1781881388281187 us; error 38 us; skew 500 ppm
I20260619 15:03:08.281584 29157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:08 dist-test-slave-zpbw krb5kdc[29052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881388, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:08.285039 29157 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:03:08.286240 29157 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:08.286372 29157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:08.286692 29157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:08.289825 29157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "308c8a87ec7f414a82bf6df57364b3e3"
format_stamp: "Formatted at 2026-06-19 15:03:08 on dist-test-slave-zpbw"
server_key: "937ef745ebb5fededbc2b16269baccf3"
server_key_iv: "611acca8bf337827e36f1b601f1620a2"
server_key_version: "encryptionkey@0"
I20260619 15:03:08.290630 29157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "308c8a87ec7f414a82bf6df57364b3e3"
format_stamp: "Formatted at 2026-06-19 15:03:08 on dist-test-slave-zpbw"
server_key: "937ef745ebb5fededbc2b16269baccf3"
server_key_iv: "611acca8bf337827e36f1b601f1620a2"
server_key_version: "encryptionkey@0"
I20260619 15:03:08.295789 29157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 15:03:08.299356 29171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:08.300833 29157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 15:03:08.300976 29157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "308c8a87ec7f414a82bf6df57364b3e3"
format_stamp: "Formatted at 2026-06-19 15:03:08 on dist-test-slave-zpbw"
server_key: "937ef745ebb5fededbc2b16269baccf3"
server_key_iv: "611acca8bf337827e36f1b601f1620a2"
server_key_version: "encryptionkey@0"
I20260619 15:03:08.301174 29157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:08.319315 29157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:08.323505 29157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:08.323742 29157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:08.324699 29157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:08.326013 29157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:08.326102 29157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:08.326203 29157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:08.326256 29157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:08.360625 29157 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:46075
I20260619 15:03:08.360685 29283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:46075 every 8 connection(s)
I20260619 15:03:08.361503 29157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:03:08.371642 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 29157
I20260619 15:03:08.371928 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:03:08.372762 24989 external_mini_cluster.cc:1428] Setting key b954dd6fc19fd4f4f1e89b484390e6d9
Jun 19 15:03:08 dist-test-slave-zpbw krb5kdc[29052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881388, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:08.384188 29286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:08.364107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:33207 (local address 127.24.103.126:35255)
0619 15:03:08.364544 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:08.364549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:08.365556 (+ 1007us) server_negotiation.cc:407] Connection header received
0619 15:03:08.367535 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:08.367541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:08.367667 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:08.367790 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:08.369390 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:08.370175 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:08.371963 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:08.372647 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:08.372827 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:08.376588 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:08.376597 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:08.376605 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:08.376664 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:08.380364 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:08.381349 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:08.381355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:08.381361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:08.381530 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:08.381967 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:08.381984 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:08.381988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:08.382375 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:08.382493 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:08.383581 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0619 15:03:08.383846 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":212,"threads_started":1}
I20260619 15:03:08.385553 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:08.364698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35255 (local address 127.24.103.65:33207)
0619 15:03:08.365384 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:08.365428 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:08.367234 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:08.368026 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:08.368034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:08.368556 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:08.369201 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:08.369218 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:08.370356 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:08.370366 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:08.371752 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:08.371764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:08.372934 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:08.372940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:08.373134 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:08.373941 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:08.373966 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:08.376335 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:08.380646 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:08.380655 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:08.380670 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:08.381177 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:08.381647 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:08.381651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:08.381653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:08.381827 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:08.382517 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:08.382529 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:08.383065 (+ 536us) client_negotiation.cc:769] Sending connection context
0619 15:03:08.383410 (+ 345us) client_negotiation.cc:240] Negotiation successful
0619 15:03:08.384023 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":280,"threads_started":1}
I20260619 15:03:08.387290 29284 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35255
I20260619 15:03:08.387652 29284 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:08.388331 29284 heartbeater.cc:511] Master 127.24.103.126:35255 requested a full tablet report, sending...
I20260619 15:03:08.390676 29096 ts_manager.cc:194] Registered new tserver with Master: 308c8a87ec7f414a82bf6df57364b3e3 (127.24.103.65:46075)
I20260619 15:03:08.392267 29096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:33207
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:03:08.419888 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35255
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:08.753786 29290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:08.754180 29290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:08.754354 29290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:08.762162 29290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:08.762336 29290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:08.762522 29290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:03:08.773290 29290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:08.774683 29290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:08.776065 29290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:08.789989 29298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:08.790169 29296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:08.790678 29295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:08.791488 29290 server_base.cc:1034] running on GCE node
I20260619 15:03:08.792153 29290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:08.793190 29290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:08.794420 29290 hybrid_clock.cc:648] HybridClock initialized: now 1781881388794373 us; error 33 us; skew 500 ppm
I20260619 15:03:08.794860 29290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:08 dist-test-slave-zpbw krb5kdc[29052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881388, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:08.798107 29290 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:03:08.799448 29290 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:08.799719 29290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:08.800128 29290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:08.803282 29290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "3436ce7c7e6d4eb58898c0e7274f3372"
format_stamp: "Formatted at 2026-06-19 15:03:08 on dist-test-slave-zpbw"
server_key: "74dc0fb9dd2bea3fbb3cf10934991877"
server_key_iv: "faae4177fbe32a097f723c38317f3108"
server_key_version: "encryptionkey@0"
I20260619 15:03:08.804191 29290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "3436ce7c7e6d4eb58898c0e7274f3372"
format_stamp: "Formatted at 2026-06-19 15:03:08 on dist-test-slave-zpbw"
server_key: "74dc0fb9dd2bea3fbb3cf10934991877"
server_key_iv: "faae4177fbe32a097f723c38317f3108"
server_key_version: "encryptionkey@0"
I20260619 15:03:08.809862 29290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 15:03:08.813603 29304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:08.815212 29290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 15:03:08.815400 29290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "3436ce7c7e6d4eb58898c0e7274f3372"
format_stamp: "Formatted at 2026-06-19 15:03:08 on dist-test-slave-zpbw"
server_key: "74dc0fb9dd2bea3fbb3cf10934991877"
server_key_iv: "faae4177fbe32a097f723c38317f3108"
server_key_version: "encryptionkey@0"
I20260619 15:03:08.815647 29290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:08.839747 29290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:08.844190 29290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:08.844506 29290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:08.845510 29290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:08.847014 29290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:08.847148 29290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:08.847292 29290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:08.847385 29290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:08.884076 29290 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:42779
I20260619 15:03:08.884220 29416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:42779 every 8 connection(s)
I20260619 15:03:08.885217 29290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:03:08.890678 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 29290
I20260619 15:03:08.890913 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:03:08.891511 24989 external_mini_cluster.cc:1428] Setting key 5ef62593f701c0159116db231eb3325d
Jun 19 15:03:08 dist-test-slave-zpbw krb5kdc[29052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881388, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:08.909480 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:08.888140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:48279 (local address 127.24.103.126:35255)
0619 15:03:08.888591 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:08.888596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:08.889615 (+ 1019us) server_negotiation.cc:407] Connection header received
0619 15:03:08.892211 (+ 2596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:08.892218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:08.892311 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:08.892425 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:08.894439 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:08.895267 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:08.897830 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:08.898248 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:08.898339 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:08.902572 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:08.902577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:08.902581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:08.902615 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:08.905830 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:08.906911 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:08.906916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:08.906918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:08.907055 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:08.907593 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:08.907597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:08.907599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:08.907872 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:08.907982 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:08.908989 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0619 15:03:08.909178 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":226,"threads_started":1}
I20260619 15:03:08.911180 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:08.888751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35255 (local address 127.24.103.66:48279)
0619 15:03:08.889440 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:08.889477 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:08.891813 (+ 2336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:08.892749 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:08.892759 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:08.893386 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:08.894241 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:08.894270 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:08.895424 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:08.895432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:08.897669 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:08.897687 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:08.898487 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:08.898504 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:08.898866 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:08.899687 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:08.899720 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:08.902284 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:08.906002 (+ 3718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:08.906015 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:08.906039 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:08.906756 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:08.907173 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:08.907180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:08.907186 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:08.907462 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:08.908007 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:08.908018 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:08.908555 (+ 537us) client_negotiation.cc:769] Sending connection context
0619 15:03:08.908915 (+ 360us) client_negotiation.cc:240] Negotiation successful
0619 15:03:08.909577 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":292,"threads_started":1}
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260619 15:03:08.912871 29417 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35255
I20260619 15:03:08.913224 29417 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:08.913844 29417 heartbeater.cc:511] Master 127.24.103.126:35255 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
I20260619 15:03:08.915483 29096 ts_manager.cc:194] Registered new tserver with Master: 3436ce7c7e6d4eb58898c0e7274f3372 (127.24.103.66:42779)
I20260619 15:03:08.916349 29096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:48279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:03:08.938405 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35255
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:09.269871 29423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:09.270248 29423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:09.270355 29423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:09.277968 29423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:09.278083 29423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:09.278206 29423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:03:09.288367 29423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:09.289641 29423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:09.290962 29423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:09.304450 29429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:09.304396 29431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:09.304422 29428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:09.305302 29423 server_base.cc:1034] running on GCE node
I20260619 15:03:09.306051 29423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:09.307130 29423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:09.308388 29423 hybrid_clock.cc:648] HybridClock initialized: now 1781881389308335 us; error 44 us; skew 500 ppm
I20260619 15:03:09.308768 29423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:09 dist-test-slave-zpbw krb5kdc[29052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881389, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:09.311957 29423 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:03:09.313150 29423 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:09.313299 29423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:09.313688 29423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:09.316802 29423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "cec4017446894711807c3cef0facb999"
format_stamp: "Formatted at 2026-06-19 15:03:09 on dist-test-slave-zpbw"
server_key: "cee30f6b6543402572a46300ef653d73"
server_key_iv: "0bcd3801c7f778947e1de954932951f5"
server_key_version: "encryptionkey@0"
I20260619 15:03:09.317685 29423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "cec4017446894711807c3cef0facb999"
format_stamp: "Formatted at 2026-06-19 15:03:09 on dist-test-slave-zpbw"
server_key: "cee30f6b6543402572a46300ef653d73"
server_key_iv: "0bcd3801c7f778947e1de954932951f5"
server_key_version: "encryptionkey@0"
I20260619 15:03:09.323379 29423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 15:03:09.327198 29437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:09.328621 29423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:03:09.328824 29423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "cec4017446894711807c3cef0facb999"
format_stamp: "Formatted at 2026-06-19 15:03:09 on dist-test-slave-zpbw"
server_key: "cee30f6b6543402572a46300ef653d73"
server_key_iv: "0bcd3801c7f778947e1de954932951f5"
server_key_version: "encryptionkey@0"
I20260619 15:03:09.329070 29423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:09.359872 29423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:09.365361 29423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:09.365674 29423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:09.366880 29423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:09.368511 29423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:09.368602 29423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:09.368680 29423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:09.368801 29423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:09.395641 29284 heartbeater.cc:503] Master 127.24.103.126:35255 was elected leader, sending a full tablet report...
I20260619 15:03:09.403774 29423 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:46783
I20260619 15:03:09.403829 29549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:46783 every 8 connection(s)
I20260619 15:03:09.404771 29423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:03:09.409345 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 29423
I20260619 15:03:09.409581 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:03:09.410012 24989 external_mini_cluster.cc:1428] Setting key e4c925414f696a0f588e492ac54f1759
Jun 19 15:03:09 dist-test-slave-zpbw krb5kdc[29052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881389, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:09.426761 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.407401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:48489 (local address 127.24.103.126:35255)
0619 15:03:09.407582 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:09.407586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:09.408777 (+ 1191us) server_negotiation.cc:407] Connection header received
0619 15:03:09.410889 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:09.410895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:09.411039 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:09.411167 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:09.413192 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.414156 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.415795 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.416211 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.416320 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.419977 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:09.419983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:09.419987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:09.420025 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:09.423149 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:09.423949 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:09.423956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:09.423959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:09.424131 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:09.424631 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:09.424636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:09.424639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:09.425172 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:09.425332 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:09.426264 (+ 932us) server_negotiation.cc:299] Negotiation successful
0619 15:03:09.426537 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 15:03:09.427798 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.407913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35255 (local address 127.24.103.67:48489)
0619 15:03:09.408603 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:09.408653 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:09.410462 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:09.411469 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:09.411479 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:09.412163 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:09.412952 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:09.412977 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.414288 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.414292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:09.415660 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:09.415672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.416397 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.416401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:09.416584 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.417262 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:09.417286 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:09.419708 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:09.423305 (+ 3597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:09.423313 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:09.423327 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:09.423798 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:09.424265 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:09.424269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:09.424272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:09.424520 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:09.425306 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:09.425313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:09.425792 (+ 479us) client_negotiation.cc:769] Sending connection context
0619 15:03:09.426122 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 15:03:09.426623 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":291,"threads_started":1}
I20260619 15:03:09.429167 29550 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35255
I20260619 15:03:09.429461 29550 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:09.430050 29550 heartbeater.cc:511] Master 127.24.103.126:35255 requested a full tablet report, sending...
I20260619 15:03:09.431444 29096 ts_manager.cc:194] Registered new tserver with Master: cec4017446894711807c3cef0facb999 (127.24.103.67:46783)
I20260619 15:03:09.432287 29096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:48489
I20260619 15:03:09.438663 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:03:09.456631 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.443234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.24.103.126:35255)
0619 15:03:09.443404 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:09.443409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:09.443973 (+ 564us) server_negotiation.cc:407] Connection header received
0619 15:03:09.444125 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:09.444128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:09.444213 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:09.444316 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:09.446014 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.446738 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.448604 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.449013 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.449106 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.451529 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:09.451533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:09.451536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:09.451570 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:09.454108 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:09.454767 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:09.454771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:09.454821 (+ 50us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:09.454963 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:09.455369 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:09.455372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:09.455374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:09.455689 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:09.455871 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:09.456227 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 15:03:09.456390 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 15:03:09.460843 29096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:03:09.468412 29096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:03:09.495249 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.481253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:46075 (local address 127.0.0.1:47202)
0619 15:03:09.482212 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:09.482263 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:09.482489 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:09.485247 (+ 2758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:09.485258 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:09.485341 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:09.486381 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:09.486401 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.489270 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.489281 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:09.492591 (+ 3310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:09.492619 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.494027 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.494033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:09.494226 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.494286 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 15:03:09.494436 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 15:03:09.494665 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":395,"spinlock_wait_cycles":3456,"thread_start_us":1586,"threads_started":1}
I20260619 15:03:09.495529 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.481555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.24.103.65:46075)
0619 15:03:09.484514 (+ 2959us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:09.484523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:09.484552 (+ 29us) server_negotiation.cc:407] Connection header received
0619 15:03:09.484653 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:09.484661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:09.484935 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:09.485825 (+ 890us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:09.486646 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.488964 (+ 2318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.492812 (+ 3848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.493800 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.493981 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.494228 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:09.494926 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 15:03:09.495132 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2785,"thread_start_us":2639,"threads_started":1}
I20260619 15:03:09.497234 29562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.481340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:46783 (local address 127.0.0.1:36044)
0619 15:03:09.483111 (+ 1771us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:09.483146 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:09.483285 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:09.485246 (+ 1961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:09.485258 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:09.485362 (+ 104us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:09.487379 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:09.487401 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.491466 (+ 4065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.491472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:09.493827 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:09.493844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.496222 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.496228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:09.496485 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.496522 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 15:03:09.496622 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 15:03:09.496802 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1635,"mutex_wait_us":91,"spinlock_wait_cycles":72448,"thread_start_us":795,"threads_started":1}
I20260619 15:03:09.497254 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.481817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.24.103.67:46783)
0619 15:03:09.484473 (+ 2656us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:09.484483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:09.484519 (+ 36us) server_negotiation.cc:407] Connection header received
0619 15:03:09.484631 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:09.484639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:09.484934 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:09.487640 (+ 2706us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:09.488010 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.491271 (+ 3261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.494052 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.495994 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.496148 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.496403 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:09.496699 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 15:03:09.496888 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2467,"thread_start_us":2718,"threads_started":1}
I20260619 15:03:09.499933 29219 tablet_service.cc:1467] Processing CreateTablet for tablet 58954cf369ad421b9555b709a7c922a6 (DEFAULT_TABLE table=test-table [id=17faec1a86494afc9831486f35df26f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:09.500763 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.484249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:42779 (local address 127.0.0.1:35196)
0619 15:03:09.491048 (+ 6799us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:09.491067 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:09.491193 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:09.492128 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:09.492133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:09.492162 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:09.492691 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:09.492739 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.494697 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.494703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:09.497996 (+ 3293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:09.498012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.499707 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.499715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:09.499909 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.499951 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 15:03:09.500054 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 15:03:09.500210 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6641,"mutex_wait_us":11,"thread_start_us":6481,"threads_started":1}
I20260619 15:03:09.500823 29567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.484697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.24.103.66:42779)
0619 15:03:09.491588 (+ 6891us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:09.491597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:09.491635 (+ 38us) server_negotiation.cc:407] Connection header received
0619 15:03:09.491716 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:09.491722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:09.491923 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:09.492152 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:09.492925 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.494475 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.498219 (+ 3744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.499459 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.499653 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.499872 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:09.500306 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 15:03:09.500485 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6738,"thread_start_us":6650,"threads_started":1}
I20260619 15:03:09.501554 29219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58954cf369ad421b9555b709a7c922a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:09.501488 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 58954cf369ad421b9555b709a7c922a6 (DEFAULT_TABLE table=test-table [id=17faec1a86494afc9831486f35df26f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:09.503400 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58954cf369ad421b9555b709a7c922a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:09.505131 29352 tablet_service.cc:1467] Processing CreateTablet for tablet 58954cf369ad421b9555b709a7c922a6 (DEFAULT_TABLE table=test-table [id=17faec1a86494afc9831486f35df26f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:09.506512 29352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58954cf369ad421b9555b709a7c922a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:09.514708 29568 tablet_bootstrap.cc:492] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999: Bootstrap starting.
I20260619 15:03:09.517760 29569 tablet_bootstrap.cc:492] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3: Bootstrap starting.
I20260619 15:03:09.519197 29568 tablet_bootstrap.cc:654] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:09.520550 29568 log.cc:824] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:09.521378 29570 tablet_bootstrap.cc:492] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372: Bootstrap starting.
I20260619 15:03:09.521435 29569 tablet_bootstrap.cc:654] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:09.523037 29569 log.cc:824] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:09.525229 29570 tablet_bootstrap.cc:654] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:09.526568 29570 log.cc:824] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:09.526615 29568 tablet_bootstrap.cc:492] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999: No bootstrap required, opened a new log
I20260619 15:03:09.526741 29569 tablet_bootstrap.cc:492] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3: No bootstrap required, opened a new log
I20260619 15:03:09.527000 29568 ts_tablet_manager.cc:1397] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260619 15:03:09.527064 29569 ts_tablet_manager.cc:1397] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 15:03:09.529911 29570 tablet_bootstrap.cc:492] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372: No bootstrap required, opened a new log
I20260619 15:03:09.530225 29570 ts_tablet_manager.cc:1397] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260619 15:03:09.534934 29568 raft_consensus.cc:348] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c8a87ec7f414a82bf6df57364b3e3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46075 } } peers { permanent_uuid: "cec4017446894711807c3cef0facb999" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46783 } } peers { permanent_uuid: "3436ce7c7e6d4eb58898c0e7274f3372" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42779 } }
I20260619 15:03:09.535354 29568 raft_consensus.cc:374] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:09.535452 29568 raft_consensus.cc:729] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec4017446894711807c3cef0facb999, State: Initialized, Role: FOLLOWER
I20260619 15:03:09.536203 29568 consensus_queue.cc:260] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c8a87ec7f414a82bf6df57364b3e3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46075 } } peers { permanent_uuid: "cec4017446894711807c3cef0facb999" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46783 } } peers { permanent_uuid: "3436ce7c7e6d4eb58898c0e7274f3372" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42779 } }
I20260619 15:03:09.538302 29569 raft_consensus.cc:348] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c8a87ec7f414a82bf6df57364b3e3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46075 } } peers { permanent_uuid: "cec4017446894711807c3cef0facb999" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46783 } } peers { permanent_uuid: "3436ce7c7e6d4eb58898c0e7274f3372" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42779 } }
I20260619 15:03:09.538841 29569 raft_consensus.cc:374] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:09.538973 29569 raft_consensus.cc:729] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308c8a87ec7f414a82bf6df57364b3e3, State: Initialized, Role: FOLLOWER
I20260619 15:03:09.539682 29550 heartbeater.cc:503] Master 127.24.103.126:35255 was elected leader, sending a full tablet report...
I20260619 15:03:09.539682 29569 consensus_queue.cc:260] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c8a87ec7f414a82bf6df57364b3e3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46075 } } peers { permanent_uuid: "cec4017446894711807c3cef0facb999" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46783 } } peers { permanent_uuid: "3436ce7c7e6d4eb58898c0e7274f3372" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42779 } }
I20260619 15:03:09.541781 29569 ts_tablet_manager.cc:1428] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260619 15:03:09.541605 29570 raft_consensus.cc:348] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c8a87ec7f414a82bf6df57364b3e3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46075 } } peers { permanent_uuid: "cec4017446894711807c3cef0facb999" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46783 } } peers { permanent_uuid: "3436ce7c7e6d4eb58898c0e7274f3372" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42779 } }
I20260619 15:03:09.542115 29570 raft_consensus.cc:374] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:09.542235 29570 raft_consensus.cc:729] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3436ce7c7e6d4eb58898c0e7274f3372, State: Initialized, Role: FOLLOWER
I20260619 15:03:09.543045 29570 consensus_queue.cc:260] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c8a87ec7f414a82bf6df57364b3e3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46075 } } peers { permanent_uuid: "cec4017446894711807c3cef0facb999" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46783 } } peers { permanent_uuid: "3436ce7c7e6d4eb58898c0e7274f3372" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42779 } }
I20260619 15:03:09.543431 29568 ts_tablet_manager.cc:1428] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260619 15:03:09.544571 29417 heartbeater.cc:503] Master 127.24.103.126:35255 was elected leader, sending a full tablet report...
I20260619 15:03:09.545055 29570 ts_tablet_manager.cc:1428] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
W20260619 15:03:09.613812 29285 tablet.cc:2369] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:03:09.633430 29574 raft_consensus.cc:482] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:03:09.633778 29574 raft_consensus.cc:504] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c8a87ec7f414a82bf6df57364b3e3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46075 } } peers { permanent_uuid: "cec4017446894711807c3cef0facb999" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46783 } } peers { permanent_uuid: "3436ce7c7e6d4eb58898c0e7274f3372" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42779 } }
I20260619 15:03:09.635803 29574 leader_election.cc:288] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 308c8a87ec7f414a82bf6df57364b3e3 (127.24.103.65:46075), 3436ce7c7e6d4eb58898c0e7274f3372 (127.24.103.66:42779)
W20260619 15:03:09.637413 29418 tablet.cc:2369] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:03:09.643503 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.636200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:46075 (local address 127.24.103.67:53937)
0619 15:03:09.636389 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:09.636410 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:09.636545 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:09.637365 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:09.637369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:09.637396 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:09.637606 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:09.637620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.639004 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.639012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:09.641018 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:09.641031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.642682 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.642692 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:09.642920 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.642957 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 15:03:09.643059 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 15:03:09.643199 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":1}
I20260619 15:03:09.643882 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.636273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:53937 (local address 127.24.103.65:46075)
0619 15:03:09.637049 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:09.637057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:09.637079 (+ 22us) server_negotiation.cc:407] Connection header received
0619 15:03:09.637163 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:09.637172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:09.637230 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:09.637345 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:09.637802 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.638865 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.641267 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.642528 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.642684 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.642764 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:09.643098 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 15:03:09.643280 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600}
I20260619 15:03:09.645121 29239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58954cf369ad421b9555b709a7c922a6" candidate_uuid: "cec4017446894711807c3cef0facb999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "308c8a87ec7f414a82bf6df57364b3e3" is_pre_election: true
I20260619 15:03:09.645419 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.636634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:42779 (local address 127.24.103.67:51017)
0619 15:03:09.637304 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:09.637323 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:09.637477 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:09.638401 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:09.638408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:09.638446 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:09.638881 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:09.638894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.640349 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.640356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:09.642854 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:09.642872 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.644549 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.644554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:09.644749 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.644782 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 15:03:09.644867 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 15:03:09.645014 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":1,"thread_start_us":1775,"threads_started":1}
I20260619 15:03:09.645777 29239 raft_consensus.cc:2454] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec4017446894711807c3cef0facb999 in term 0.
I20260619 15:03:09.646749 29567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.637879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:51017 (local address 127.24.103.66:42779)
0619 15:03:09.638067 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:09.638075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:09.638094 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:03:09.638172 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:09.638185 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:09.638255 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:09.638343 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:09.639132 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.640210 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.643038 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.644356 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.644541 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.646298 (+ 1757us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:09.646354 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 15:03:09.646476 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 15:03:09.646665 29440 leader_election.cc:302] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 308c8a87ec7f414a82bf6df57364b3e3, cec4017446894711807c3cef0facb999; no voters:
I20260619 15:03:09.647217 29574 raft_consensus.cc:2781] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:03:09.647374 29574 raft_consensus.cc:482] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:03:09.647500 29574 raft_consensus.cc:3037] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:09.647894 29372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58954cf369ad421b9555b709a7c922a6" candidate_uuid: "cec4017446894711807c3cef0facb999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3436ce7c7e6d4eb58898c0e7274f3372" is_pre_election: true
I20260619 15:03:09.648516 29372 raft_consensus.cc:2454] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec4017446894711807c3cef0facb999 in term 0.
I20260619 15:03:09.650892 29574 raft_consensus.cc:504] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c8a87ec7f414a82bf6df57364b3e3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46075 } } peers { permanent_uuid: "cec4017446894711807c3cef0facb999" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46783 } } peers { permanent_uuid: "3436ce7c7e6d4eb58898c0e7274f3372" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42779 } }
I20260619 15:03:09.651716 29574 leader_election.cc:288] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [CANDIDATE]: Term 1 election: Requested vote from peers 308c8a87ec7f414a82bf6df57364b3e3 (127.24.103.65:46075), 3436ce7c7e6d4eb58898c0e7274f3372 (127.24.103.66:42779)
I20260619 15:03:09.652328 29372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58954cf369ad421b9555b709a7c922a6" candidate_uuid: "cec4017446894711807c3cef0facb999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3436ce7c7e6d4eb58898c0e7274f3372"
I20260619 15:03:09.652354 29239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58954cf369ad421b9555b709a7c922a6" candidate_uuid: "cec4017446894711807c3cef0facb999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "308c8a87ec7f414a82bf6df57364b3e3"
I20260619 15:03:09.652547 29372 raft_consensus.cc:3037] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:09.652606 29239 raft_consensus.cc:3037] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:09.654870 29372 raft_consensus.cc:2454] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec4017446894711807c3cef0facb999 in term 1.
I20260619 15:03:09.654942 29239 raft_consensus.cc:2454] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec4017446894711807c3cef0facb999 in term 1.
I20260619 15:03:09.655519 29440 leader_election.cc:302] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 308c8a87ec7f414a82bf6df57364b3e3, cec4017446894711807c3cef0facb999; no voters:
I20260619 15:03:09.655870 29574 raft_consensus.cc:2781] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:09.656378 29574 raft_consensus.cc:686] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 1 LEADER]: Becoming Leader. State: Replica: cec4017446894711807c3cef0facb999, State: Running, Role: LEADER
W20260619 15:03:09.656561 29551 tablet.cc:2369] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:03:09.656989 29574 consensus_queue.cc:237] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c8a87ec7f414a82bf6df57364b3e3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46075 } } peers { permanent_uuid: "cec4017446894711807c3cef0facb999" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46783 } } peers { permanent_uuid: "3436ce7c7e6d4eb58898c0e7274f3372" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42779 } }
I20260619 15:03:09.661700 29094 catalog_manager.cc:5495] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec4017446894711807c3cef0facb999 (127.24.103.67). New cstate: current_term: 1 leader_uuid: "cec4017446894711807c3cef0facb999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c8a87ec7f414a82bf6df57364b3e3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec4017446894711807c3cef0facb999" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3436ce7c7e6d4eb58898c0e7274f3372" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42779 } health_report { overall_health: UNKNOWN } } }
I20260619 15:03:09.692094 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.686638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.24.103.67:46783)
0619 15:03:09.686931 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:09.686939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:09.687066 (+ 127us) server_negotiation.cc:407] Connection header received
0619 15:03:09.687244 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:09.687248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:09.687336 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:09.687528 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:03:09.687860 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.688642 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.689937 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.690344 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.690447 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.690953 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:03:09.691512 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:03:09.691648 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:09.691711 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 15:03:09.691782 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 15:03:09.700441 29372 raft_consensus.cc:1261] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372 [term 1 FOLLOWER]: Refusing update from remote peer cec4017446894711807c3cef0facb999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:09.700441 29239 raft_consensus.cc:1261] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3 [term 1 FOLLOWER]: Refusing update from remote peer cec4017446894711807c3cef0facb999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:09.701483 29578 consensus_queue.cc:1035] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3436ce7c7e6d4eb58898c0e7274f3372" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:09.701923 29574 consensus_queue.cc:1035] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "308c8a87ec7f414a82bf6df57364b3e3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:09.731400 29094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:09.731717 29094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:09.736593 29094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ab9270032e842c8869fdf89ba580a7e: Sending DeleteTablet for 3 replicas of tablet 58954cf369ad421b9555b709a7c922a6
I20260619 15:03:09.737923 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 58954cf369ad421b9555b709a7c922a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:09 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:36044
I20260619 15:03:09.737932 29219 tablet_service.cc:1514] Processing DeleteTablet for tablet 58954cf369ad421b9555b709a7c922a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:09 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:47202
I20260619 15:03:09.739117 29591 tablet_replica.cc:330] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3: stopping tablet replica
I20260619 15:03:09.739838 29591 raft_consensus.cc:2229] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:09.740624 29591 raft_consensus.cc:2258] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:09.743781 29591 ts_tablet_manager.cc:1905] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:09.745088 29352 tablet_service.cc:1514] Processing DeleteTablet for tablet 58954cf369ad421b9555b709a7c922a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:09 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:35196
I20260619 15:03:09.751348 29592 tablet_replica.cc:330] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999: stopping tablet replica
I20260619 15:03:09.751866 29592 raft_consensus.cc:2229] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:03:09.752651 29592 raft_consensus.cc:2258] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:09.754094 29591 ts_tablet_manager.cc:1918] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:09.754407 29591 log.cc:1196] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/58954cf369ad421b9555b709a7c922a6
I20260619 15:03:09.755151 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 29157
I20260619 15:03:09.755168 29592 ts_tablet_manager.cc:1905] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:09.755219 29591 ts_tablet_manager.cc:1939] T 58954cf369ad421b9555b709a7c922a6 P 308c8a87ec7f414a82bf6df57364b3e3: Deleting consensus metadata
I20260619 15:03:09.757692 29082 catalog_manager.cc:4841] TS 308c8a87ec7f414a82bf6df57364b3e3 (127.24.103.65:46075): tablet 58954cf369ad421b9555b709a7c922a6 (table test-table [id=17faec1a86494afc9831486f35df26f3]) successfully deleted
I20260619 15:03:09.763685 29593 tablet_replica.cc:330] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372: stopping tablet replica
I20260619 15:03:09.764477 29593 raft_consensus.cc:2229] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:09.765264 29593 raft_consensus.cc:2258] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:09.767421 29592 ts_tablet_manager.cc:1918] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:09.767623 29592 log.cc:1196] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/58954cf369ad421b9555b709a7c922a6
I20260619 15:03:09.768301 29592 ts_tablet_manager.cc:1939] T 58954cf369ad421b9555b709a7c922a6 P cec4017446894711807c3cef0facb999: Deleting consensus metadata
I20260619 15:03:09.768393 29593 ts_tablet_manager.cc:1905] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:09.770251 29083 catalog_manager.cc:4841] TS cec4017446894711807c3cef0facb999 (127.24.103.67:46783): tablet 58954cf369ad421b9555b709a7c922a6 (table test-table [id=17faec1a86494afc9831486f35df26f3]) successfully deleted
I20260619 15:03:09.775568 29593 ts_tablet_manager.cc:1918] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:09.775782 29593 log.cc:1196] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/58954cf369ad421b9555b709a7c922a6
I20260619 15:03:09.776430 29593 ts_tablet_manager.cc:1939] T 58954cf369ad421b9555b709a7c922a6 P 3436ce7c7e6d4eb58898c0e7274f3372: Deleting consensus metadata
Jun 19 15:03:09 dist-test-slave-zpbw krb5kdc[29052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:03:09.778496 29082 catalog_manager.cc:4841] TS 3436ce7c7e6d4eb58898c0e7274f3372 (127.24.103.66:42779): tablet 58954cf369ad421b9555b709a7c922a6 (table test-table [id=17faec1a86494afc9831486f35df26f3]) successfully deleted
I20260619 15:03:09.791818 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.757030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.24.103.65:46075)
0619 15:03:09.757222 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:09.757229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:09.766288 (+ 9059us) server_negotiation.cc:407] Connection header received
0619 15:03:09.766626 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:09.766634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:09.766737 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:09.766849 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:09.769190 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.770349 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.772528 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.773200 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.773354 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.779432 (+ 6078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:09.779447 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:09.779462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:09.779503 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:09.788645 (+ 9142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:09.789714 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:09.789722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:09.789724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:09.789866 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:09.790329 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:09.790332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:09.790334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:09.790694 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:09.791014 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:09.791320 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 15:03:09.791495 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 15:03:09.792598 29279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:09.883527 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 29157
I20260619 15:03:09.907402 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 29290
Jun 19 15:03:09 dist-test-slave-zpbw krb5kdc[29052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:03:09.923720 29567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:09.908294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35202 (local address 127.24.103.66:42779)
0619 15:03:09.908470 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:09.908477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:09.908499 (+ 22us) server_negotiation.cc:407] Connection header received
0619 15:03:09.908734 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:09.908741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:09.908816 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:09.908923 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:09.910168 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.910999 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.912279 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:09.912700 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:09.912845 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:09.917245 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:09.917261 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:09.917276 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:09.917313 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:09.920767 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:09.921522 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:09.921530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:09.921532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:09.921662 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:09.922069 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:09.922076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:09.922079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:09.922451 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:09.922754 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:09.923078 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 15:03:09.923290 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 15:03:09.925047 29412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:10.016325 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 29290
I20260619 15:03:10.034132 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 29423
Jun 19 15:03:10 dist-test-slave-zpbw krb5kdc[29052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:03:10.051769 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:10.034959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.24.103.67:46783)
0619 15:03:10.035157 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:10.035165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:10.035205 (+ 40us) server_negotiation.cc:407] Connection header received
0619 15:03:10.035392 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:10.035399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:10.035494 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:10.035620 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:10.036846 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:10.038088 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:10.039548 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:10.040276 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:10.040423 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:10.045387 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:10.045405 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:10.045420 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:10.045468 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:10.048576 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:10.049586 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:10.049593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:10.049595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:10.049758 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:10.050212 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:10.050215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:10.050217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:10.050565 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:10.050962 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:10.051225 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 15:03:10.051394 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 15:03:10.052587 29545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:10.174578 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 29423
I20260619 15:03:10.193836 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 29065
I20260619 15:03:10.195034 29126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:10.303718 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 29065
2026-06-19T15:03:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4359 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T15:03:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:03:10Z Disabled control of system clock
I20260619 15:03:10.372120 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38457
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 15:03:10.704768 29602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:10.705108 29602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:10.705193 29602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:10.713807 29602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:10.713923 29602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:10.713963 29602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:10.714023 29602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:10.714069 29602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:03:10.723665 29602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:10.724957 29602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:10.726712 29602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:10.739121 29607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:10.739090 29608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:10.739141 29610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:10.739961 29602 server_base.cc:1034] running on GCE node
I20260619 15:03:10.740815 29602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:10.742327 29602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:10.743584 29602 hybrid_clock.cc:648] HybridClock initialized: now 1781881390743538 us; error 69 us; skew 500 ppm
I20260619 15:03:10.743960 29602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:03:10.746327 29602 webserver.cc:460] Webserver started at http://127.24.103.126:38275/ using document root <none> and password file <none>
I20260619 15:03:10.747159 29602 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:10.747294 29602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:10.747617 29602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:10.750559 29602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "f74e7239e4b3453b94b55ceb0f8f1d14"
format_stamp: "Formatted at 2026-06-19 15:03:10 on dist-test-slave-zpbw"
server_key: "2e077785b54d37f7e69a55b7d36f044b"
server_key_iv: "945071fb0c8a7087680c91bd4950b794"
server_key_version: "encryptionkey@0"
I20260619 15:03:10.751543 29602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "f74e7239e4b3453b94b55ceb0f8f1d14"
format_stamp: "Formatted at 2026-06-19 15:03:10 on dist-test-slave-zpbw"
server_key: "2e077785b54d37f7e69a55b7d36f044b"
server_key_iv: "945071fb0c8a7087680c91bd4950b794"
server_key_version: "encryptionkey@0"
I20260619 15:03:10.756906 29602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:03:10.760682 29616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:10.762200 29602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 15:03:10.762403 29602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "f74e7239e4b3453b94b55ceb0f8f1d14"
format_stamp: "Formatted at 2026-06-19 15:03:10 on dist-test-slave-zpbw"
server_key: "2e077785b54d37f7e69a55b7d36f044b"
server_key_iv: "945071fb0c8a7087680c91bd4950b794"
server_key_version: "encryptionkey@0"
I20260619 15:03:10.762655 29602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:10.768940 29602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:10.769754 29602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:10.770085 29602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:10.789114 29602 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:38457
I20260619 15:03:10.789149 29667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:38457 every 8 connection(s)
I20260619 15:03:10.790498 29602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:10.792486 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 29602
I20260619 15:03:10.792689 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:03:10.793398 24989 external_mini_cluster.cc:1428] Setting key 042d5daf9f671dddccb07f9df9452e61
I20260619 15:03:10.794766 29668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:10.801828 29671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:10.795472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.24.103.126:38457)
0619 15:03:10.796527 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:10.796538 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:10.796578 (+ 40us) server_negotiation.cc:407] Connection header received
0619 15:03:10.798079 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:10.798100 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:10.798518 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:10.798947 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:10.800069 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:10.800074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:10.800087 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:10.800101 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:10.800173 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:10.800189 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:10.800457 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:10.800604 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 15:03:10.800889 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":448,"threads_started":1}
I20260619 15:03:10.807106 29668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14: Bootstrap starting.
I20260619 15:03:10.810175 29668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:10.811353 29668 log.cc:824] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:10.814024 29668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14: No bootstrap required, opened a new log
I20260619 15:03:10.821522 29668 raft_consensus.cc:348] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74e7239e4b3453b94b55ceb0f8f1d14" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38457 } }
I20260619 15:03:10.821900 29668 raft_consensus.cc:374] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:10.821969 29668 raft_consensus.cc:729] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e7239e4b3453b94b55ceb0f8f1d14, State: Initialized, Role: FOLLOWER
I20260619 15:03:10.822611 29668 consensus_queue.cc:260] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74e7239e4b3453b94b55ceb0f8f1d14" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38457 } }
I20260619 15:03:10.822908 29668 raft_consensus.cc:388] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:10.823055 29668 raft_consensus.cc:482] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:10.823242 29668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:10.825704 29668 raft_consensus.cc:504] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74e7239e4b3453b94b55ceb0f8f1d14" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38457 } }
I20260619 15:03:10.826236 29668 leader_election.cc:302] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f74e7239e4b3453b94b55ceb0f8f1d14; no voters:
I20260619 15:03:10.826887 29668 leader_election.cc:288] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:03:10.827044 29673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:10.827781 29673 raft_consensus.cc:686] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [term 1 LEADER]: Becoming Leader. State: Replica: f74e7239e4b3453b94b55ceb0f8f1d14, State: Running, Role: LEADER
I20260619 15:03:10.828325 29673 consensus_queue.cc:237] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74e7239e4b3453b94b55ceb0f8f1d14" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38457 } }
I20260619 15:03:10.829093 29668 sys_catalog.cc:564] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:10.831230 29675 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f74e7239e4b3453b94b55ceb0f8f1d14. Latest consensus state: current_term: 1 leader_uuid: "f74e7239e4b3453b94b55ceb0f8f1d14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74e7239e4b3453b94b55ceb0f8f1d14" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38457 } } }
I20260619 15:03:10.831542 29675 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:10.832815 29674 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f74e7239e4b3453b94b55ceb0f8f1d14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74e7239e4b3453b94b55ceb0f8f1d14" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38457 } } }
I20260619 15:03:10.833292 29674 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:10.835358 29678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:10.841113 29678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:10.850740 29678 catalog_manager.cc:1347] Generated new cluster ID: 34f60e39e63c4c8aa3efb118c2c8e3c7
I20260619 15:03:10.850950 29678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:10.867611 29678 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:03:10.868070 29678 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:10.875644 29678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14: Generated new TSK 0
I20260619 15:03:10.876489 29678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:03:10.890688 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38457
--builtin_ntp_servers=127.24.103.84:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 15:03:11.232434 29692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:11.232740 29692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:11.232887 29692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:11.240736 29692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:11.240859 29692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:11.241037 29692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:03:11.250831 29692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:11.252295 29692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:11.254256 29692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:11.267854 29697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:11.267906 29700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:11.267823 29698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:11.268920 29692 server_base.cc:1034] running on GCE node
I20260619 15:03:11.269541 29692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:11.270502 29692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:11.271703 29692 hybrid_clock.cc:648] HybridClock initialized: now 1781881391271654 us; error 43 us; skew 500 ppm
I20260619 15:03:11.272100 29692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:03:11.274533 29692 webserver.cc:460] Webserver started at http://127.24.103.65:32955/ using document root <none> and password file <none>
I20260619 15:03:11.275409 29692 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:11.275503 29692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:11.275851 29692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:11.278861 29692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "95f4ba863fc14324b53dc9dad62bf36b"
format_stamp: "Formatted at 2026-06-19 15:03:11 on dist-test-slave-zpbw"
server_key: "4359d046141ff9b2ef9b6d3681042d0d"
server_key_iv: "335a28a517c6cfb354787115f4913551"
server_key_version: "encryptionkey@0"
I20260619 15:03:11.279721 29692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "95f4ba863fc14324b53dc9dad62bf36b"
format_stamp: "Formatted at 2026-06-19 15:03:11 on dist-test-slave-zpbw"
server_key: "4359d046141ff9b2ef9b6d3681042d0d"
server_key_iv: "335a28a517c6cfb354787115f4913551"
server_key_version: "encryptionkey@0"
I20260619 15:03:11.285336 29692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 15:03:11.289059 29706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:11.290515 29692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:03:11.290730 29692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "95f4ba863fc14324b53dc9dad62bf36b"
format_stamp: "Formatted at 2026-06-19 15:03:11 on dist-test-slave-zpbw"
server_key: "4359d046141ff9b2ef9b6d3681042d0d"
server_key_iv: "335a28a517c6cfb354787115f4913551"
server_key_version: "encryptionkey@0"
I20260619 15:03:11.291025 29692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:11.300606 29692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:11.301743 29692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:11.301978 29692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:11.303062 29692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:11.304451 29692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:11.304535 29692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:11.304611 29692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:11.304673 29692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:11.339437 29692 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:38089
I20260619 15:03:11.339555 29818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:38089 every 8 connection(s)
I20260619 15:03:11.340864 29692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:03:11.349030 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 29692
I20260619 15:03:11.349238 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:03:11.349877 24989 external_mini_cluster.cc:1428] Setting key 6973fa6c3e35d398c5b1471cab2e0727
I20260619 15:03:11.350140 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:11.343933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:40375 (local address 127.24.103.126:38457)
0619 15:03:11.344488 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:11.344493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:11.345809 (+ 1316us) server_negotiation.cc:407] Connection header received
0619 15:03:11.347650 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:11.347656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:11.347727 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:11.347859 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:11.349310 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:11.349316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:11.349321 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:11.349343 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:11.349395 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:11.349410 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:11.349571 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:11.349759 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 15:03:11.349837 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":292,"threads_started":1}
I20260619 15:03:11.351402 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:11.344581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38457 (local address 127.24.103.65:40375)
0619 15:03:11.345635 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:11.345676 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:11.347365 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:11.348120 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:11.348129 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:11.348226 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:11.348963 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:03:11.348966 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:11.349004 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:03:11.349092 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:11.349525 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:11.349535 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:11.349545 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 15:03:11.349847 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 15:03:11.350128 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":294,"threads_started":1}
I20260619 15:03:11.353122 29819 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38457
I20260619 15:03:11.353350 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38457
--builtin_ntp_servers=127.24.103.84:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 15:03:11.353508 29819 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:11.354116 29819 heartbeater.cc:511] Master 127.24.103.126:38457 requested a full tablet report, sending...
I20260619 15:03:11.356585 29633 ts_manager.cc:194] Registered new tserver with Master: 95f4ba863fc14324b53dc9dad62bf36b (127.24.103.65:38089)
W20260619 15:03:11.691179 29823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:11.691540 29823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:11.691637 29823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:11.699525 29823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:11.699641 29823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:11.699795 29823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:03:11.709964 29823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:11.711359 29823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:11.713712 29823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:11.728514 29831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:11.728510 29829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:11.728502 29828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:11.729120 29823 server_base.cc:1034] running on GCE node
I20260619 15:03:11.729893 29823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:11.731019 29823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:11.732230 29823 hybrid_clock.cc:648] HybridClock initialized: now 1781881391732190 us; error 37 us; skew 500 ppm
I20260619 15:03:11.732582 29823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:03:11.734965 29823 webserver.cc:460] Webserver started at http://127.24.103.66:42785/ using document root <none> and password file <none>
I20260619 15:03:11.735714 29823 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:11.735831 29823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:11.736133 29823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:11.739166 29823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "ef3f817fce984300a1418d6f3c2eea3e"
format_stamp: "Formatted at 2026-06-19 15:03:11 on dist-test-slave-zpbw"
server_key: "c9937497cd3903f77c9980ed7fb1e314"
server_key_iv: "d5cf4d9bba8114cbf5612a4c90eac8bb"
server_key_version: "encryptionkey@0"
I20260619 15:03:11.739923 29823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "ef3f817fce984300a1418d6f3c2eea3e"
format_stamp: "Formatted at 2026-06-19 15:03:11 on dist-test-slave-zpbw"
server_key: "c9937497cd3903f77c9980ed7fb1e314"
server_key_iv: "d5cf4d9bba8114cbf5612a4c90eac8bb"
server_key_version: "encryptionkey@0"
I20260619 15:03:11.745379 29823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 15:03:11.749323 29837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:11.751020 29823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:03:11.751226 29823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "ef3f817fce984300a1418d6f3c2eea3e"
format_stamp: "Formatted at 2026-06-19 15:03:11 on dist-test-slave-zpbw"
server_key: "c9937497cd3903f77c9980ed7fb1e314"
server_key_iv: "d5cf4d9bba8114cbf5612a4c90eac8bb"
server_key_version: "encryptionkey@0"
I20260619 15:03:11.751505 29823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:11.757870 29823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:11.758844 29823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:11.759148 29823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:11.760499 29823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:11.762140 29823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:11.762285 29823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:11.762442 29823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:11.762540 29823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:11.797937 29823 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:43829
I20260619 15:03:11.798039 29949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:43829 every 8 connection(s)
I20260619 15:03:11.799360 29823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:03:11.805368 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 29823
I20260619 15:03:11.805593 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:03:11.806054 24989 external_mini_cluster.cc:1428] Setting key e3b95ebde71329dd56b3aac7559bc93e
I20260619 15:03:11.809152 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38457
--builtin_ntp_servers=127.24.103.84:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 15:03:11.809664 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:11.802975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:60079 (local address 127.24.103.126:38457)
0619 15:03:11.803193 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:11.803200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:11.804279 (+ 1079us) server_negotiation.cc:407] Connection header received
0619 15:03:11.806739 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:11.806747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:11.806849 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:11.806980 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:11.808728 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:11.808735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:11.808742 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:11.808766 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:11.808809 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:11.808828 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:11.808961 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:11.809193 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 15:03:11.809262 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 15:03:11.810703 29952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:11.802700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38457 (local address 127.24.103.66:60079)
0619 15:03:11.804101 (+ 1401us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:11.804148 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:11.806397 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:11.807247 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:11.807256 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:11.807356 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:11.808336 (+ 980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:03:11.808341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:11.808395 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:03:11.808511 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:11.808938 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:11.808947 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:11.808955 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 15:03:11.809248 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 15:03:11.809584 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":473,"threads_started":1}
I20260619 15:03:11.812372 29950 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38457
I20260619 15:03:11.812723 29950 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:11.813331 29950 heartbeater.cc:511] Master 127.24.103.126:38457 requested a full tablet report, sending...
I20260619 15:03:11.814852 29633 ts_manager.cc:194] Registered new tserver with Master: ef3f817fce984300a1418d6f3c2eea3e (127.24.103.66:43829)
W20260619 15:03:12.136969 29953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:12.137262 29953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:12.137411 29953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:12.144878 29953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:12.144994 29953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:12.145161 29953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:03:12.154438 29953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:12.155871 29953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:12.157651 29953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:12.171067 29959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:12.171023 29958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:12.171181 29961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:12.171756 29953 server_base.cc:1034] running on GCE node
I20260619 15:03:12.172331 29953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:12.173323 29953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:12.174521 29953 hybrid_clock.cc:648] HybridClock initialized: now 1781881392174482 us; error 48 us; skew 500 ppm
I20260619 15:03:12.174858 29953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:03:12.177026 29953 webserver.cc:460] Webserver started at http://127.24.103.67:42323/ using document root <none> and password file <none>
I20260619 15:03:12.177673 29953 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:12.177759 29953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:12.178017 29953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:12.180969 29953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "72e3c6ed36ba4e2faae336f654e8edc9"
format_stamp: "Formatted at 2026-06-19 15:03:12 on dist-test-slave-zpbw"
server_key: "d8ce0a866dc76d4216e73fd4182d1ef9"
server_key_iv: "f95e3d9bd1d481dedecae4952e15537e"
server_key_version: "encryptionkey@0"
I20260619 15:03:12.181710 29953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "72e3c6ed36ba4e2faae336f654e8edc9"
format_stamp: "Formatted at 2026-06-19 15:03:12 on dist-test-slave-zpbw"
server_key: "d8ce0a866dc76d4216e73fd4182d1ef9"
server_key_iv: "f95e3d9bd1d481dedecae4952e15537e"
server_key_version: "encryptionkey@0"
I20260619 15:03:12.186755 29953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:03:12.190250 29967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:12.191655 29953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 15:03:12.191795 29953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "72e3c6ed36ba4e2faae336f654e8edc9"
format_stamp: "Formatted at 2026-06-19 15:03:12 on dist-test-slave-zpbw"
server_key: "d8ce0a866dc76d4216e73fd4182d1ef9"
server_key_iv: "f95e3d9bd1d481dedecae4952e15537e"
server_key_version: "encryptionkey@0"
I20260619 15:03:12.191979 29953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:12.197651 29953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:12.198380 29953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:12.198596 29953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:12.199563 29953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:12.200871 29953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:12.200954 29953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:12.201026 29953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:12.201083 29953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:12.233960 29953 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:38435
I20260619 15:03:12.234041 30079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:38435 every 8 connection(s)
I20260619 15:03:12.235395 29953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:03:12.239902 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 29953
I20260619 15:03:12.240064 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:03:12.240545 24989 external_mini_cluster.cc:1428] Setting key f2e420ac47ed47683ccd15fe320734d3
I20260619 15:03:12.244365 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.238124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:50161 (local address 127.24.103.126:38457)
0619 15:03:12.238314 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:12.238318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:12.239835 (+ 1517us) server_negotiation.cc:407] Connection header received
0619 15:03:12.241939 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:12.241945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:12.242022 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:12.242130 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:12.243675 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:12.243680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:12.243685 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:12.243707 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:12.243757 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:12.243775 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:12.243891 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:12.244102 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 15:03:12.244154 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:03:12.245430 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.238638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38457 (local address 127.24.103.67:50161)
0619 15:03:12.239647 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.239688 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:12.241591 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:12.242373 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:12.242386 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:12.242548 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:12.243309 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:03:12.243312 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:12.243361 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:03:12.243493 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:12.243876 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:12.243884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:12.243890 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 15:03:12.244146 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 15:03:12.244354 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":339,"threads_started":1}
I20260619 15:03:12.246845 30080 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38457
I20260619 15:03:12.247159 30080 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:12.247762 30080 heartbeater.cc:511] Master 127.24.103.126:38457 requested a full tablet report, sending...
I20260619 15:03:12.249194 29633 ts_manager.cc:194] Registered new tserver with Master: 72e3c6ed36ba4e2faae336f654e8edc9 (127.24.103.67:38435)
I20260619 15:03:12.256289 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:03:12.263556 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.260974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.24.103.126:38457)
0619 15:03:12.261225 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:12.261233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:12.261556 (+ 323us) server_negotiation.cc:407] Connection header received
0619 15:03:12.261704 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:12.261708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:12.261761 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:12.261846 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:12.262886 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:12.262891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:12.262895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:12.262913 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:12.262954 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:12.262972 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:12.263118 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:12.263199 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 15:03:12.263248 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:03:12.267876 29633 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:03:12.274854 29633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:03:12.305356 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.287205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:43829 (local address 127.0.0.1:38936)
0619 15:03:12.301008 (+ 13803us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.301065 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:12.301398 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:12.302603 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:12.302610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:12.302719 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:12.303647 (+ 928us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:03:12.303651 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:12.303680 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:03:12.303771 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:12.304465 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:12.304478 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:12.304489 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 15:03:12.304674 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 15:03:12.304783 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13412,"thread_start_us":13150,"threads_started":1}
I20260619 15:03:12.305864 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.288040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.24.103.67:38435)
0619 15:03:12.299784 (+ 11744us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:12.299798 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:12.302687 (+ 2889us) server_negotiation.cc:407] Connection header received
0619 15:03:12.302997 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:12.303008 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:12.303212 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:12.303339 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:12.304843 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:12.304850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:12.304862 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:12.304881 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:12.304937 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:12.304956 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:12.305094 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:12.305373 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 15:03:12.305481 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11447,"thread_start_us":11768,"threads_started":1}
I20260619 15:03:12.306196 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.287474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.24.103.66:43829)
0619 15:03:12.301985 (+ 14511us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:12.301991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:12.302018 (+ 27us) server_negotiation.cc:407] Connection header received
0619 15:03:12.302120 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:12.302128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:12.302384 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:12.302576 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:12.303944 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:12.303954 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:12.303971 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:12.304000 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:12.304077 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:12.304100 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:12.305631 (+ 1531us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:12.305703 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 15:03:12.305797 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14347,"thread_start_us":14224,"threads_started":1}
I20260619 15:03:12.307147 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.287431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:38435 (local address 127.0.0.1:36426)
0619 15:03:12.302563 (+ 15132us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.302582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:12.302816 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:12.303831 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:12.303838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:12.303862 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:12.304587 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:03:12.304592 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:12.304607 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:03:12.304660 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:12.305197 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:12.305203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:12.305208 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 15:03:12.306695 (+ 1487us) client_negotiation.cc:240] Negotiation successful
0619 15:03:12.306748 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14973,"spinlock_wait_cycles":6400,"thread_start_us":14959,"threads_started":1}
I20260619 15:03:12.309523 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.304314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38089 (local address 127.0.0.1:38048)
0619 15:03:12.304983 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.305004 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:12.305114 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:12.307707 (+ 2593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:12.307714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:12.307739 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:12.308402 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:03:12.308405 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:12.308423 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:03:12.308472 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:12.308830 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:12.308835 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:12.308840 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 15:03:12.308981 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 15:03:12.309044 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":2258,"threads_started":1}
I20260619 15:03:12.310168 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.303154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.24.103.65:38089)
0619 15:03:12.307184 (+ 4030us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:12.307191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:12.307216 (+ 25us) server_negotiation.cc:407] Connection header received
0619 15:03:12.307269 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:12.307275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:12.307491 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:12.307715 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:12.308588 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:12.308598 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:12.308610 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:12.308643 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:12.308693 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:12.308710 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:12.308840 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:12.309758 (+ 918us) server_negotiation.cc:299] Negotiation successful
0619 15:03:12.309852 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3877,"thread_start_us":3750,"threads_started":1}
I20260619 15:03:12.310562 29885 tablet_service.cc:1467] Processing CreateTablet for tablet 8bad16c7207b44649ab953ccba1daf74 (DEFAULT_TABLE table=test-table [id=a0d36359687c490c9fcf189521279b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:12.311151 30015 tablet_service.cc:1467] Processing CreateTablet for tablet 8bad16c7207b44649ab953ccba1daf74 (DEFAULT_TABLE table=test-table [id=a0d36359687c490c9fcf189521279b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:12.312107 29885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bad16c7207b44649ab953ccba1daf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:12.312577 30015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bad16c7207b44649ab953ccba1daf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:12.313599 29754 tablet_service.cc:1467] Processing CreateTablet for tablet 8bad16c7207b44649ab953ccba1daf74 (DEFAULT_TABLE table=test-table [id=a0d36359687c490c9fcf189521279b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:12.314940 29754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bad16c7207b44649ab953ccba1daf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:12.323770 30098 tablet_bootstrap.cc:492] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e: Bootstrap starting.
I20260619 15:03:12.327147 30099 tablet_bootstrap.cc:492] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9: Bootstrap starting.
I20260619 15:03:12.327919 30098 tablet_bootstrap.cc:654] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:12.329284 30098 log.cc:824] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:12.329596 30100 tablet_bootstrap.cc:492] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b: Bootstrap starting.
I20260619 15:03:12.331018 30099 tablet_bootstrap.cc:654] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:12.332916 30100 tablet_bootstrap.cc:654] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:12.333498 30099 log.cc:824] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:12.334100 30100 log.cc:824] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:12.336917 30099 tablet_bootstrap.cc:492] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9: No bootstrap required, opened a new log
I20260619 15:03:12.336948 30098 tablet_bootstrap.cc:492] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e: No bootstrap required, opened a new log
I20260619 15:03:12.337215 30098 ts_tablet_manager.cc:1397] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260619 15:03:12.337236 30099 ts_tablet_manager.cc:1397] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260619 15:03:12.338649 30100 tablet_bootstrap.cc:492] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b: No bootstrap required, opened a new log
I20260619 15:03:12.338999 30100 ts_tablet_manager.cc:1397] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 15:03:12.346431 30099 raft_consensus.cc:348] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3f817fce984300a1418d6f3c2eea3e" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43829 } } peers { permanent_uuid: "72e3c6ed36ba4e2faae336f654e8edc9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38435 } } peers { permanent_uuid: "95f4ba863fc14324b53dc9dad62bf36b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38089 } }
I20260619 15:03:12.346966 30099 raft_consensus.cc:374] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:12.347122 30099 raft_consensus.cc:729] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e3c6ed36ba4e2faae336f654e8edc9, State: Initialized, Role: FOLLOWER
I20260619 15:03:12.347275 30098 raft_consensus.cc:348] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3f817fce984300a1418d6f3c2eea3e" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43829 } } peers { permanent_uuid: "72e3c6ed36ba4e2faae336f654e8edc9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38435 } } peers { permanent_uuid: "95f4ba863fc14324b53dc9dad62bf36b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38089 } }
I20260619 15:03:12.347760 30098 raft_consensus.cc:374] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:12.347883 30098 raft_consensus.cc:729] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3f817fce984300a1418d6f3c2eea3e, State: Initialized, Role: FOLLOWER
I20260619 15:03:12.347841 30099 consensus_queue.cc:260] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3f817fce984300a1418d6f3c2eea3e" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43829 } } peers { permanent_uuid: "72e3c6ed36ba4e2faae336f654e8edc9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38435 } } peers { permanent_uuid: "95f4ba863fc14324b53dc9dad62bf36b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38089 } }
I20260619 15:03:12.348568 30098 consensus_queue.cc:260] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3f817fce984300a1418d6f3c2eea3e" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43829 } } peers { permanent_uuid: "72e3c6ed36ba4e2faae336f654e8edc9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38435 } } peers { permanent_uuid: "95f4ba863fc14324b53dc9dad62bf36b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38089 } }
I20260619 15:03:12.355062 30100 raft_consensus.cc:348] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3f817fce984300a1418d6f3c2eea3e" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43829 } } peers { permanent_uuid: "72e3c6ed36ba4e2faae336f654e8edc9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38435 } } peers { permanent_uuid: "95f4ba863fc14324b53dc9dad62bf36b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38089 } }
I20260619 15:03:12.355731 30100 raft_consensus.cc:374] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:12.355844 30100 raft_consensus.cc:729] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f4ba863fc14324b53dc9dad62bf36b, State: Initialized, Role: FOLLOWER
I20260619 15:03:12.356837 30098 ts_tablet_manager.cc:1428] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
I20260619 15:03:12.356705 30100 consensus_queue.cc:260] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3f817fce984300a1418d6f3c2eea3e" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43829 } } peers { permanent_uuid: "72e3c6ed36ba4e2faae336f654e8edc9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38435 } } peers { permanent_uuid: "95f4ba863fc14324b53dc9dad62bf36b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38089 } }
I20260619 15:03:12.357502 29950 heartbeater.cc:503] Master 127.24.103.126:38457 was elected leader, sending a full tablet report...
I20260619 15:03:12.357964 30080 heartbeater.cc:503] Master 127.24.103.126:38457 was elected leader, sending a full tablet report...
I20260619 15:03:12.358726 30099 ts_tablet_manager.cc:1428] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9: Time spent starting tablet: real 0.021s user 0.009s sys 0.014s
I20260619 15:03:12.360821 29819 heartbeater.cc:503] Master 127.24.103.126:38457 was elected leader, sending a full tablet report...
I20260619 15:03:12.367776 30100 ts_tablet_manager.cc:1428] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b: Time spent starting tablet: real 0.029s user 0.016s sys 0.007s
W20260619 15:03:12.487164 30081 tablet.cc:2369] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:03:12.514454 30105 raft_consensus.cc:482] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:03:12.514876 30105 raft_consensus.cc:504] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3f817fce984300a1418d6f3c2eea3e" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43829 } } peers { permanent_uuid: "72e3c6ed36ba4e2faae336f654e8edc9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38435 } } peers { permanent_uuid: "95f4ba863fc14324b53dc9dad62bf36b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38089 } }
I20260619 15:03:12.516619 30105 leader_election.cc:288] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72e3c6ed36ba4e2faae336f654e8edc9 (127.24.103.67:38435), 95f4ba863fc14324b53dc9dad62bf36b (127.24.103.65:38089)
I20260619 15:03:12.520190 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.517034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:41721 (local address 127.24.103.67:38435)
0619 15:03:12.517255 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:12.517263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:12.517889 (+ 626us) server_negotiation.cc:407] Connection header received
0619 15:03:12.518113 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:12.518120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:12.518200 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:12.518276 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:12.519563 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:12.519569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:12.519578 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:12.519592 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:12.519653 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:12.519668 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:12.519804 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:12.519851 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 15:03:12.519910 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:03:12.520334 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.518126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:45355 (local address 127.24.103.65:38089)
0619 15:03:12.518306 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:12.518314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:12.518536 (+ 222us) server_negotiation.cc:407] Connection header received
0619 15:03:12.518607 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:12.518614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:12.518679 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:12.518830 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:12.519715 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:12.519724 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:12.519730 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:12.519751 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:12.519787 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:12.519801 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:12.519910 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:12.520029 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 15:03:12.520084 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:03:12.520433 30107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.517040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:38435 (local address 127.24.103.66:41721)
0619 15:03:12.517708 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.517725 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:12.517916 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:12.518480 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:12.518510 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:12.518535 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:12.519338 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:03:12.519340 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:12.519353 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:03:12.519389 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:12.519760 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:12.519763 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:12.519766 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 15:03:12.519887 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 15:03:12.519938 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":372,"threads_started":1}
I20260619 15:03:12.520536 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.517813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38089 (local address 127.24.103.66:45355)
0619 15:03:12.518426 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.518444 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:12.518527 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:12.518868 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:12.518871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:12.518886 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:12.519533 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:03:12.519536 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:12.519548 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:03:12.519581 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:12.519912 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:12.519916 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:12.519921 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 15:03:12.520054 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 15:03:12.520101 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":314,"threads_started":1}
I20260619 15:03:12.521616 30035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bad16c7207b44649ab953ccba1daf74" candidate_uuid: "ef3f817fce984300a1418d6f3c2eea3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e3c6ed36ba4e2faae336f654e8edc9" is_pre_election: true
I20260619 15:03:12.521653 29774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bad16c7207b44649ab953ccba1daf74" candidate_uuid: "ef3f817fce984300a1418d6f3c2eea3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95f4ba863fc14324b53dc9dad62bf36b" is_pre_election: true
I20260619 15:03:12.522132 30035 raft_consensus.cc:2454] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3f817fce984300a1418d6f3c2eea3e in term 0.
I20260619 15:03:12.522171 29774 raft_consensus.cc:2454] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3f817fce984300a1418d6f3c2eea3e in term 0.
I20260619 15:03:12.522856 29840 leader_election.cc:302] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72e3c6ed36ba4e2faae336f654e8edc9, ef3f817fce984300a1418d6f3c2eea3e; no voters:
I20260619 15:03:12.523303 30105 raft_consensus.cc:2781] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:03:12.523491 30105 raft_consensus.cc:482] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:03:12.523617 30105 raft_consensus.cc:3037] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:12.525882 30105 raft_consensus.cc:504] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3f817fce984300a1418d6f3c2eea3e" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43829 } } peers { permanent_uuid: "72e3c6ed36ba4e2faae336f654e8edc9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38435 } } peers { permanent_uuid: "95f4ba863fc14324b53dc9dad62bf36b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38089 } }
I20260619 15:03:12.526486 30105 leader_election.cc:288] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [CANDIDATE]: Term 1 election: Requested vote from peers 72e3c6ed36ba4e2faae336f654e8edc9 (127.24.103.67:38435), 95f4ba863fc14324b53dc9dad62bf36b (127.24.103.65:38089)
I20260619 15:03:12.526985 30035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bad16c7207b44649ab953ccba1daf74" candidate_uuid: "ef3f817fce984300a1418d6f3c2eea3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e3c6ed36ba4e2faae336f654e8edc9"
I20260619 15:03:12.527086 29774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bad16c7207b44649ab953ccba1daf74" candidate_uuid: "ef3f817fce984300a1418d6f3c2eea3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95f4ba863fc14324b53dc9dad62bf36b"
I20260619 15:03:12.527206 30035 raft_consensus.cc:3037] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:12.527333 29774 raft_consensus.cc:3037] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:12.529572 30035 raft_consensus.cc:2454] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3f817fce984300a1418d6f3c2eea3e in term 1.
I20260619 15:03:12.529593 29774 raft_consensus.cc:2454] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3f817fce984300a1418d6f3c2eea3e in term 1.
I20260619 15:03:12.530017 29840 leader_election.cc:302] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72e3c6ed36ba4e2faae336f654e8edc9, ef3f817fce984300a1418d6f3c2eea3e; no voters:
I20260619 15:03:12.530382 30105 raft_consensus.cc:2781] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:12.530959 30105 raft_consensus.cc:686] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 1 LEADER]: Becoming Leader. State: Replica: ef3f817fce984300a1418d6f3c2eea3e, State: Running, Role: LEADER
I20260619 15:03:12.531514 30105 consensus_queue.cc:237] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3f817fce984300a1418d6f3c2eea3e" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43829 } } peers { permanent_uuid: "72e3c6ed36ba4e2faae336f654e8edc9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38435 } } peers { permanent_uuid: "95f4ba863fc14324b53dc9dad62bf36b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38089 } }
I20260619 15:03:12.536520 29631 catalog_manager.cc:5495] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e reported cstate change: term changed from 0 to 1, leader changed from <none> to ef3f817fce984300a1418d6f3c2eea3e (127.24.103.66). New cstate: current_term: 1 leader_uuid: "ef3f817fce984300a1418d6f3c2eea3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3f817fce984300a1418d6f3c2eea3e" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72e3c6ed36ba4e2faae336f654e8edc9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95f4ba863fc14324b53dc9dad62bf36b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20260619 15:03:12.549672 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.547642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.24.103.66:43829)
0619 15:03:12.547836 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:12.547844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:12.547862 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:03:12.547966 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:12.547971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:12.548047 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:12.548166 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:12.549043 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:12.549049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:12.549053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:12.549071 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:12.549107 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:12.549119 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:12.549242 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:12.549355 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 15:03:12.549413 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 15:03:12.551365 29951 tablet.cc:2369] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:03:12.558521 30035 raft_consensus.cc:1261] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9 [term 1 FOLLOWER]: Refusing update from remote peer ef3f817fce984300a1418d6f3c2eea3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:12.558521 29774 raft_consensus.cc:1261] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b [term 1 FOLLOWER]: Refusing update from remote peer ef3f817fce984300a1418d6f3c2eea3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:12.559623 30105 consensus_queue.cc:1035] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e3c6ed36ba4e2faae336f654e8edc9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:12.559981 30109 consensus_queue.cc:1035] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f4ba863fc14324b53dc9dad62bf36b" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:12.588693 29633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:12.589015 29633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:12.592983 29633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f74e7239e4b3453b94b55ceb0f8f1d14: Sending DeleteTablet for 3 replicas of tablet 8bad16c7207b44649ab953ccba1daf74
I20260619 15:03:12.594297 30015 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bad16c7207b44649ab953ccba1daf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:12 UTC) from {username='slave'} at 127.0.0.1:36426
I20260619 15:03:12.594344 29885 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bad16c7207b44649ab953ccba1daf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:12 UTC) from {username='slave'} at 127.0.0.1:38936
I20260619 15:03:12.595351 30122 tablet_replica.cc:330] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e: stopping tablet replica
I20260619 15:03:12.595633 30123 tablet_replica.cc:330] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9: stopping tablet replica
I20260619 15:03:12.595724 29754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bad16c7207b44649ab953ccba1daf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:12 UTC) from {username='slave'} at 127.0.0.1:38048
I20260619 15:03:12.596138 30122 raft_consensus.cc:2229] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:03:12.596117 30123 raft_consensus.cc:2229] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:12.596788 30123 raft_consensus.cc:2258] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:12.596985 30122 raft_consensus.cc:2258] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:12.598433 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 29692
I20260619 15:03:12.599753 30122 ts_tablet_manager.cc:1905] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:12.607460 30124 tablet_replica.cc:330] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b: stopping tablet replica
I20260619 15:03:12.607944 30124 raft_consensus.cc:2229] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:12.608481 30124 raft_consensus.cc:2258] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:12.608575 30123 ts_tablet_manager.cc:1905] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:12.609735 30122 ts_tablet_manager.cc:1918] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:12.609978 30122 log.cc:1196] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/8bad16c7207b44649ab953ccba1daf74
I20260619 15:03:12.610553 30124 ts_tablet_manager.cc:1905] T 8bad16c7207b44649ab953ccba1daf74 P 95f4ba863fc14324b53dc9dad62bf36b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:12.610622 30122 ts_tablet_manager.cc:1939] T 8bad16c7207b44649ab953ccba1daf74 P ef3f817fce984300a1418d6f3c2eea3e: Deleting consensus metadata
I20260619 15:03:12.612558 29620 catalog_manager.cc:4841] TS ef3f817fce984300a1418d6f3c2eea3e (127.24.103.66:43829): tablet 8bad16c7207b44649ab953ccba1daf74 (table test-table [id=a0d36359687c490c9fcf189521279b07]) successfully deleted
I20260619 15:03:12.614102 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.601481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.24.103.65:38089)
0619 15:03:12.601743 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:12.601753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:12.611380 (+ 9627us) server_negotiation.cc:407] Connection header received
0619 15:03:12.611732 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:12.611738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:12.611832 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:12.611929 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:12.613327 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:12.613333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:12.613338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:12.613356 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:12.613392 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:12.613407 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:12.613530 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:12.613750 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 15:03:12.613818 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 15:03:12.615397 29814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:12.617267 30123 ts_tablet_manager.cc:1918] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:12.617591 30123 log.cc:1196] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/8bad16c7207b44649ab953ccba1daf74
I20260619 15:03:12.618538 30123 ts_tablet_manager.cc:1939] T 8bad16c7207b44649ab953ccba1daf74 P 72e3c6ed36ba4e2faae336f654e8edc9: Deleting consensus metadata
I20260619 15:03:12.621059 29619 catalog_manager.cc:4841] TS 72e3c6ed36ba4e2faae336f654e8edc9 (127.24.103.67:38435): tablet 8bad16c7207b44649ab953ccba1daf74 (table test-table [id=a0d36359687c490c9fcf189521279b07]) successfully deleted
I20260619 15:03:12.705046 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 29692
W20260619 15:03:12.723510 29619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.103.65:38089 (error 108)
I20260619 15:03:12.723790 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 29823
I20260619 15:03:12.725746 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.725154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38089 (local address 127.0.0.1:38054)
0619 15:03:12.725389 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.725563 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260619 15:03:12.726377 29619 catalog_manager.cc:4584] TS 95f4ba863fc14324b53dc9dad62bf36b (127.24.103.65:38089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bad16c7207b44649ab953ccba1daf74: Network error: Client connection negotiation failed: client connection to 127.24.103.65:38089: connect: Connection refused (error 111)
I20260619 15:03:12.727419 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.724815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.24.103.66:43829)
0619 15:03:12.725061 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:12.725068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:12.725091 (+ 23us) server_negotiation.cc:407] Connection header received
0619 15:03:12.725252 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:12.725259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:12.725330 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:12.725452 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:12.726496 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:12.726502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:12.726507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:12.726530 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:12.726568 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:12.726584 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:12.726714 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:12.727015 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 15:03:12.727089 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 15:03:12.728232 29945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:12.780318 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.779687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38089 (local address 127.0.0.1:38056)
0619 15:03:12.779956 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.780100 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 15:03:12.781605 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.781154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38089 (local address 127.0.0.1:38070)
0619 15:03:12.781331 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.781434 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 15:03:12.839666 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 29823
I20260619 15:03:12.842022 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.841567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38089 (local address 127.0.0.1:38084)
0619 15:03:12.841779 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.841879 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 15:03:12.843089 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.842705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38089 (local address 127.0.0.1:38098)
0619 15:03:12.842868 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.842959 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 15:03:12.858707 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 29953
I20260619 15:03:12.861990 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.859555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.24.103.67:38435)
0619 15:03:12.859756 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:12.859763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:12.859863 (+ 100us) server_negotiation.cc:407] Connection header received
0619 15:03:12.860079 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:12.860085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:12.860188 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:12.860302 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:12.861262 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:12.861268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:12.861272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:12.861291 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:12.861333 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:12.861345 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:12.861486 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:12.861637 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 15:03:12.861699 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 15:03:12.862700 30075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:12.923897 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.923246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38089 (local address 127.0.0.1:38112)
0619 15:03:12.923519 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.923666 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 15:03:12.925240 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:12.924711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38089 (local address 127.0.0.1:38118)
0619 15:03:12.924956 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:12.925062 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 15:03:12.956070 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 29953
I20260619 15:03:12.972363 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 29602
I20260619 15:03:12.973546 29663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:13.079841 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 29602
2026-06-19T15:03:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2775 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 15:03:13.121417 24989 mini_oidc.cc:152] Starting JWKS server
I20260619 15:03:13.124192 24989 webserver.cc:460] Webserver started at https://127.0.0.1:46619/ using document root <none> and password file <none>
I20260619 15:03:13.124539 24989 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 15:03:13.125284 24989 webserver.cc:460] Webserver started at http://127.0.0.1:38509/ using document root <none> and password file <none>
2026-06-19T15:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:03:13Z Disabled control of system clock
I20260619 15:03:13.153285 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43869
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:43869
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46619/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/testchainca.cert with env {}
W20260619 15:03:13.482295 30136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:13.482620 30136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:13.482710 30136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:13.490235 30136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:13.490365 30136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:13.490403 30136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:13.490453 30136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:13.490494 30136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 15:03:13.490554 30136 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 15:03:13.490592 30136 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46619/jwks.json
I20260619 15:03:13.501066 30136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:43869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46619/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:13.502540 30136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:13.503862 30136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:13.515862 30141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:13.515818 30144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:13.515883 30142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:13.517226 30136 server_base.cc:1034] running on GCE node
I20260619 15:03:13.517990 30136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:13.519462 30136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:13.520697 30136 hybrid_clock.cc:648] HybridClock initialized: now 1781881393520651 us; error 56 us; skew 500 ppm
I20260619 15:03:13.521059 30136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:03:13.523552 30136 webserver.cc:460] Webserver started at http://127.24.103.126:39645/ using document root <none> and password file <none>
I20260619 15:03:13.524269 30136 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:13.524415 30136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:13.524796 30136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:13.527926 30136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "5ce971956b8a44fab93299bcf097643c"
format_stamp: "Formatted at 2026-06-19 15:03:13 on dist-test-slave-zpbw"
server_key: "9dc0e7cf07b9d3e79ddf902d6751b6c2"
server_key_iv: "a0c5901ea968c1f22925608a14c0d2b8"
server_key_version: "encryptionkey@0"
I20260619 15:03:13.528735 30136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "5ce971956b8a44fab93299bcf097643c"
format_stamp: "Formatted at 2026-06-19 15:03:13 on dist-test-slave-zpbw"
server_key: "9dc0e7cf07b9d3e79ddf902d6751b6c2"
server_key_iv: "a0c5901ea968c1f22925608a14c0d2b8"
server_key_version: "encryptionkey@0"
I20260619 15:03:13.534251 30136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:03:13.538137 30150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:13.539724 30136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:03:13.539942 30136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "5ce971956b8a44fab93299bcf097643c"
format_stamp: "Formatted at 2026-06-19 15:03:13 on dist-test-slave-zpbw"
server_key: "9dc0e7cf07b9d3e79ddf902d6751b6c2"
server_key_iv: "a0c5901ea968c1f22925608a14c0d2b8"
server_key_version: "encryptionkey@0"
I20260619 15:03:13.540180 30136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:13.556066 30136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:13.556990 30136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:13.557351 30136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:13.576423 30136 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:43869
I20260619 15:03:13.576397 30201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:43869 every 8 connection(s)
I20260619 15:03:13.577903 30136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:13.581905 30202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:13.584030 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 30136
I20260619 15:03:13.584376 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:03:13.585083 24989 external_mini_cluster.cc:1428] Setting key b7eacde52d93f9cdb7f5ba074d7b9ce8
I20260619 15:03:13.596452 30202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c: Bootstrap starting.
I20260619 15:03:13.600396 30202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:13.601687 30202 log.cc:824] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:13.601770 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:13.587354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.24.103.126:43869)
0619 15:03:13.588522 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:13.588538 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:13.588593 (+ 55us) server_negotiation.cc:407] Connection header received
0619 15:03:13.590960 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:13.590990 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:13.591656 (+ 666us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:13.592182 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:13.592796 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:13.594920 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:13.596402 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:13.597439 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:13.597629 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:13.598843 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:13.598853 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:13.598872 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:13.599594 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:13.599722 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:13.599760 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:13.599874 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:13.600107 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 15:03:13.600457 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":349,"threads_started":1}
I20260619 15:03:13.604889 30202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c: No bootstrap required, opened a new log
I20260619 15:03:13.606241 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:03:13.613291 30202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce971956b8a44fab93299bcf097643c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43869 } }
I20260619 15:03:13.613632 30202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:13.613704 30202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce971956b8a44fab93299bcf097643c, State: Initialized, Role: FOLLOWER
I20260619 15:03:13.614202 30202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce971956b8a44fab93299bcf097643c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43869 } }
I20260619 15:03:13.614387 30202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:13.614492 30202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:13.614611 30202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:13.616854 30202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce971956b8a44fab93299bcf097643c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43869 } }
I20260619 15:03:13.617391 30202 leader_election.cc:302] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ce971956b8a44fab93299bcf097643c; no voters:
I20260619 15:03:13.617966 30202 leader_election.cc:288] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:03:13.618129 30207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:13.618875 30207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [term 1 LEADER]: Becoming Leader. State: Replica: 5ce971956b8a44fab93299bcf097643c, State: Running, Role: LEADER
I20260619 15:03:13.619343 30207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce971956b8a44fab93299bcf097643c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43869 } }
I20260619 15:03:13.620100 30202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:13.622162 30209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ce971956b8a44fab93299bcf097643c. Latest consensus state: current_term: 1 leader_uuid: "5ce971956b8a44fab93299bcf097643c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce971956b8a44fab93299bcf097643c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43869 } } }
I20260619 15:03:13.622556 30209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:13.624692 30208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ce971956b8a44fab93299bcf097643c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce971956b8a44fab93299bcf097643c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43869 } } }
I20260619 15:03:13.625046 30208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:13.625741 30213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:13.631075 30213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:13.640594 30213 catalog_manager.cc:1347] Generated new cluster ID: 1b611d51f08845b8b13354e209034359
I20260619 15:03:13.640728 30213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:13.662284 30213 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:03:13.664290 30213 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:13.674270 30213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ce971956b8a44fab93299bcf097643c: Generated new TSK 0
I20260619 15:03:13.675462 30213 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 15:03:13.720186 30232 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 15:03:13.720448 30232 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 15:03:13.723007 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:13.692501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.24.103.126:43869)
0619 15:03:13.692761 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:13.692768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:13.693217 (+ 449us) server_negotiation.cc:407] Connection header received
0619 15:03:13.693550 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:13.693554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:13.693636 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:13.693761 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0619 15:03:13.694403 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:13.695213 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:13.697420 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:13.697855 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:13.697953 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:13.698574 (+ 621us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 15:03:13.721014 (+ 22440us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46619/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 15:03:13.722498 (+ 1484us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46968: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46619/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":85}
W20260619 15:03:13.723188 30205 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46968: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46619/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 15:03:13.723649 30231 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:13.692406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:43869 (local address 127.0.0.1:46968)
0619 15:03:13.693099 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:13.693120 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:13.693323 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:13.693828 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:13.693836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:13.693853 (+ 17us) client_negotiation.cc:189] Negotiated authn=JWT
0619 15:03:13.694256 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:13.694270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:13.695365 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:13.695373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:13.697187 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:13.697203 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:13.698021 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:13.698027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:13.698207 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:13.698295 (+ 88us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 15:03:13.721381 (+ 23086us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46619/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 15:03:13.721760 (+ 379us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.103.126:43869: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46619/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":335,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.103.126:43869: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46619/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 15:03:13.725759 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 30136
I20260619 15:03:13.727052 30197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:13.821172 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 30136
2026-06-19T15:03:13Z chronyd exiting
I20260619 15:03:13.871693 24989 test_util.cc:175] -----------------------------------------------
I20260619 15:03:13.871887 24989 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881351219607-24989-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (755 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 15:03:13.876205 24989 mini_oidc.cc:152] Starting JWKS server
I20260619 15:03:13.878163 24989 webserver.cc:460] Webserver started at https://127.0.0.1:40667/ using document root <none> and password file <none>
I20260619 15:03:13.878407 24989 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 15:03:13.879079 24989 webserver.cc:460] Webserver started at http://127.0.0.1:42117/ using document root <none> and password file <none>
2026-06-19T15:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:03:13Z Disabled control of system clock
I20260619 15:03:13.906042 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34899
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:34899
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40667/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/testchainca.cert with env {}
W20260619 15:03:14.233100 30239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:14.233398 30239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:14.233485 30239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:14.240937 30239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:14.241078 30239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:14.241117 30239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:14.241178 30239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:14.241222 30239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 15:03:14.241283 30239 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 15:03:14.241323 30239 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40667/jwks.json
I20260619 15:03:14.251626 30239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:34899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40667/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:14.252910 30239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:14.254211 30239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:14.266762 30244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:14.266848 30245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:14.266875 30247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:14.267783 30239 server_base.cc:1034] running on GCE node
I20260619 15:03:14.268632 30239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:14.270164 30239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:14.271390 30239 hybrid_clock.cc:648] HybridClock initialized: now 1781881394271335 us; error 57 us; skew 500 ppm
I20260619 15:03:14.271752 30239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:03:14.274240 30239 webserver.cc:460] Webserver started at http://127.24.103.126:41901/ using document root <none> and password file <none>
I20260619 15:03:14.275053 30239 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:14.275202 30239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:14.275533 30239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:14.278611 30239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "76b3b00db52f4b1aa2cd39313685fd4c"
format_stamp: "Formatted at 2026-06-19 15:03:14 on dist-test-slave-zpbw"
server_key: "d8c2874d23be0b9af8cd203bc1d88f48"
server_key_iv: "aa49bd00bc2de064014e76875f0786cf"
server_key_version: "encryptionkey@0"
I20260619 15:03:14.279625 30239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "76b3b00db52f4b1aa2cd39313685fd4c"
format_stamp: "Formatted at 2026-06-19 15:03:14 on dist-test-slave-zpbw"
server_key: "d8c2874d23be0b9af8cd203bc1d88f48"
server_key_iv: "aa49bd00bc2de064014e76875f0786cf"
server_key_version: "encryptionkey@0"
I20260619 15:03:14.285219 30239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:03:14.289036 30253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:14.290589 30239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:03:14.290871 30239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "76b3b00db52f4b1aa2cd39313685fd4c"
format_stamp: "Formatted at 2026-06-19 15:03:14 on dist-test-slave-zpbw"
server_key: "d8c2874d23be0b9af8cd203bc1d88f48"
server_key_iv: "aa49bd00bc2de064014e76875f0786cf"
server_key_version: "encryptionkey@0"
I20260619 15:03:14.291121 30239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:14.304498 30239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:14.305459 30239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:14.305795 30239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:14.324733 30239 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:34899
I20260619 15:03:14.324739 30304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:34899 every 8 connection(s)
I20260619 15:03:14.326215 30239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:14.330145 30305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:14.336367 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 30239
I20260619 15:03:14.336620 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:03:14.337260 24989 external_mini_cluster.cc:1428] Setting key f2e8ad67099421b0d2e70a11ebf2a562
I20260619 15:03:14.341585 30305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c: Bootstrap starting.
I20260619 15:03:14.344952 30305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:14.346022 30305 log.cc:824] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:14.349105 30305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c: No bootstrap required, opened a new log
I20260619 15:03:14.351265 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:14.339393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.24.103.126:34899)
0619 15:03:14.340430 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:14.340441 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:14.340484 (+ 43us) server_negotiation.cc:407] Connection header received
0619 15:03:14.342259 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:14.342284 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:14.342766 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:14.343283 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:14.343861 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:14.345225 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:14.346581 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:14.347376 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:14.347575 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:14.348798 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:14.348805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:14.348819 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:14.349420 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:14.349506 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:14.349535 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:14.349677 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:14.349896 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 15:03:14.350207 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":348,"threads_started":1}
I20260619 15:03:14.355371 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:03:14.357990 30305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b3b00db52f4b1aa2cd39313685fd4c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34899 } }
I20260619 15:03:14.358485 30305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:14.358635 30305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b3b00db52f4b1aa2cd39313685fd4c, State: Initialized, Role: FOLLOWER
I20260619 15:03:14.359263 30305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b3b00db52f4b1aa2cd39313685fd4c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34899 } }
I20260619 15:03:14.359468 30305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:14.359565 30305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:14.359699 30305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:14.361842 30305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b3b00db52f4b1aa2cd39313685fd4c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34899 } }
I20260619 15:03:14.362358 30305 leader_election.cc:302] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76b3b00db52f4b1aa2cd39313685fd4c; no voters:
I20260619 15:03:14.363004 30305 leader_election.cc:288] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:03:14.363153 30310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:14.364620 30310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [term 1 LEADER]: Becoming Leader. State: Replica: 76b3b00db52f4b1aa2cd39313685fd4c, State: Running, Role: LEADER
I20260619 15:03:14.365216 30305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:14.365162 30310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b3b00db52f4b1aa2cd39313685fd4c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34899 } }
I20260619 15:03:14.368449 30311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76b3b00db52f4b1aa2cd39313685fd4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b3b00db52f4b1aa2cd39313685fd4c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34899 } } }
I20260619 15:03:14.368294 30312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76b3b00db52f4b1aa2cd39313685fd4c. Latest consensus state: current_term: 1 leader_uuid: "76b3b00db52f4b1aa2cd39313685fd4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b3b00db52f4b1aa2cd39313685fd4c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34899 } } }
I20260619 15:03:14.368805 30311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:14.368849 30312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:14.375176 30319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:14.380574 30319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:14.389948 30319 catalog_manager.cc:1347] Generated new cluster ID: c5fd546dbc98433caff4f398288f82ca
I20260619 15:03:14.390085 30319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:14.412158 30319 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:03:14.414104 30319 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:14.421835 30319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76b3b00db52f4b1aa2cd39313685fd4c: Generated new TSK 0
I20260619 15:03:14.422948 30319 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 15:03:14.464946 30335 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 15:03:14.465092 30335 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 15:03:14.466118 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:14.438171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44474 (local address 127.24.103.126:34899)
0619 15:03:14.438409 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:14.438416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:14.438833 (+ 417us) server_negotiation.cc:407] Connection header received
0619 15:03:14.439080 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:14.439084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:14.439159 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:14.439304 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0619 15:03:14.439866 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:14.440588 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:14.442652 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:14.443180 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:14.443282 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:14.443878 (+ 596us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 15:03:14.465514 (+ 21636us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 15:03:14.465898 (+ 384us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44474: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":97}
W20260619 15:03:14.466267 30308 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44474: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 15:03:14.466495 30334 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:14.437965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34899 (local address 127.0.0.1:44474)
0619 15:03:14.438666 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:14.438693 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:14.438886 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:14.439351 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:14.439356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:14.439370 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0619 15:03:14.439688 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:14.439698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:14.440738 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:14.440744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:14.442473 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:14.442507 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:14.443368 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:14.443375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:14.443589 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:14.443637 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 15:03:14.465786 (+ 22149us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 15:03:14.466104 (+ 318us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.103.126:34899: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":357,"threads_started":1}
I20260619 15:03:14.467617 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 30239
I20260619 15:03:14.468731 30300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:14.562939 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 30239
2026-06-19T15:03:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (739 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 15:03:14.615008 24989 mini_oidc.cc:152] Starting JWKS server
I20260619 15:03:14.616904 24989 webserver.cc:460] Webserver started at https://127.0.0.1:40371/ using document root <none> and password file <none>
I20260619 15:03:14.617110 24989 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 15:03:14.617753 24989 webserver.cc:460] Webserver started at http://127.0.0.1:37233/ using document root <none> and password file <none>
2026-06-19T15:03:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:03:14Z Disabled control of system clock
I20260619 15:03:14.646279 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:37717
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:40061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:37717
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40371/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 15:03:14.980201 30342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:14.980520 30342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:14.980618 30342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:14.988111 30342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:14.988232 30342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:14.988274 30342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:14.988336 30342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:14.988380 30342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 15:03:14.988435 30342 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 15:03:14.988476 30342 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40371/jwks.json
I20260619 15:03:14.998034 30342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:37717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:37717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40371/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:14.999357 30342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:15.001210 30342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:15.013715 30347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:15.013744 30350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:15.013769 30348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:15.014323 30342 server_base.cc:1034] running on GCE node
I20260619 15:03:15.015372 30342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:15.016887 30342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:15.018095 30342 hybrid_clock.cc:648] HybridClock initialized: now 1781881395018042 us; error 43 us; skew 500 ppm
I20260619 15:03:15.018473 30342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:03:15.024564 30342 webserver.cc:460] Webserver started at http://127.24.103.126:37519/ using document root <none> and password file <none>
I20260619 15:03:15.025385 30342 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:15.025539 30342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:15.025897 30342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:15.029271 30342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "6e252a79f7b24378877f7a7373fd0fff"
format_stamp: "Formatted at 2026-06-19 15:03:15 on dist-test-slave-zpbw"
server_key: "4dfa5feda14b1de2d9a263dd0e6d35fa"
server_key_iv: "cd01f06756d1c1baad08e448ade48c56"
server_key_version: "encryptionkey@0"
I20260619 15:03:15.030205 30342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "6e252a79f7b24378877f7a7373fd0fff"
format_stamp: "Formatted at 2026-06-19 15:03:15 on dist-test-slave-zpbw"
server_key: "4dfa5feda14b1de2d9a263dd0e6d35fa"
server_key_iv: "cd01f06756d1c1baad08e448ade48c56"
server_key_version: "encryptionkey@0"
I20260619 15:03:15.036166 30342 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260619 15:03:15.039997 30356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:15.041626 30342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:03:15.041869 30342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "6e252a79f7b24378877f7a7373fd0fff"
format_stamp: "Formatted at 2026-06-19 15:03:15 on dist-test-slave-zpbw"
server_key: "4dfa5feda14b1de2d9a263dd0e6d35fa"
server_key_iv: "cd01f06756d1c1baad08e448ade48c56"
server_key_version: "encryptionkey@0"
I20260619 15:03:15.042122 30342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:15.062388 30342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:15.063356 30342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:15.063707 30342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:15.083422 30342 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:37717
I20260619 15:03:15.083443 30407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:37717 every 8 connection(s)
I20260619 15:03:15.084995 30342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:15.087426 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 30342
I20260619 15:03:15.087656 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:03:15.088461 24989 external_mini_cluster.cc:1428] Setting key 67d075c78b6137c8f38849f724471fd0
I20260619 15:03:15.090545 30408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:15.102692 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:15.090257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.24.103.126:37717)
0619 15:03:15.091295 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:15.091306 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:15.091347 (+ 41us) server_negotiation.cc:407] Connection header received
0619 15:03:15.092997 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:15.093019 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:15.093517 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:15.093883 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:15.094498 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:15.095952 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:15.097862 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:15.098565 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:15.098703 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:15.100065 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:15.100075 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:15.100091 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:15.100721 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:15.100802 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:15.100823 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:15.100917 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:15.101175 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 15:03:15.101639 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":386,"threads_started":1}
I20260619 15:03:15.104235 30408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff: Bootstrap starting.
I20260619 15:03:15.106993 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:03:15.107910 30408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:15.109000 30408 log.cc:824] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:15.112545 30408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff: No bootstrap required, opened a new log
I20260619 15:03:15.120142 30408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e252a79f7b24378877f7a7373fd0fff" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37717 } }
I20260619 15:03:15.120627 30408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:15.120751 30408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e252a79f7b24378877f7a7373fd0fff, State: Initialized, Role: FOLLOWER
I20260619 15:03:15.121362 30408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e252a79f7b24378877f7a7373fd0fff" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37717 } }
I20260619 15:03:15.121623 30408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:15.121774 30408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:15.121963 30408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:15.124289 30408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e252a79f7b24378877f7a7373fd0fff" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37717 } }
I20260619 15:03:15.124848 30408 leader_election.cc:302] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e252a79f7b24378877f7a7373fd0fff; no voters:
I20260619 15:03:15.125470 30408 leader_election.cc:288] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:03:15.125700 30413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:15.126523 30413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [term 1 LEADER]: Becoming Leader. State: Replica: 6e252a79f7b24378877f7a7373fd0fff, State: Running, Role: LEADER
I20260619 15:03:15.127228 30413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e252a79f7b24378877f7a7373fd0fff" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37717 } }
I20260619 15:03:15.128103 30408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:15.130605 30415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e252a79f7b24378877f7a7373fd0fff. Latest consensus state: current_term: 1 leader_uuid: "6e252a79f7b24378877f7a7373fd0fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e252a79f7b24378877f7a7373fd0fff" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37717 } } }
I20260619 15:03:15.130553 30414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e252a79f7b24378877f7a7373fd0fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e252a79f7b24378877f7a7373fd0fff" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37717 } } }
I20260619 15:03:15.131032 30415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:15.131088 30414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:15.133272 30420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:15.138397 30420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:15.147696 30420 catalog_manager.cc:1347] Generated new cluster ID: a0e9b7708bc34ad28eb15fc3c3d29aad
I20260619 15:03:15.147833 30420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:15.170483 30420 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:03:15.172506 30420 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:15.180380 30420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e252a79f7b24378877f7a7373fd0fff: Generated new TSK 0
I20260619 15:03:15.181567 30420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:03:15.210906 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:15.201797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.24.103.126:37717)
0619 15:03:15.202053 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:15.202063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:15.202492 (+ 429us) server_negotiation.cc:407] Connection header received
0619 15:03:15.202754 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:15.202759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:15.202897 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:15.203037 (+ 140us) server_negotiation.cc:226] Negotiated authn=JWT
0619 15:03:15.203609 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:15.204353 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:15.206449 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:15.207070 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:15.207221 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:15.207840 (+ 619us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 15:03:15.210210 (+ 2370us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40371/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 15:03:15.210557 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51902: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40371/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 15:03:15.211030 30411 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51902: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40371/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 15:03:15.213145 30437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:15.201718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:37717 (local address 127.0.0.1:51902)
0619 15:03:15.202371 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:15.202393 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:15.202561 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:15.203085 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:15.203090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:15.203104 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0619 15:03:15.203445 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:15.203457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:15.204505 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:15.204511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:15.206278 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:15.206296 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:15.207293 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:15.207300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:15.207497 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:15.207549 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 15:03:15.210454 (+ 2905us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40371/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 15:03:15.212637 (+ 2183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.103.126:37717: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40371/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":1815,"thread_start_us":331,"threads_started":1}
I20260619 15:03:15.215190 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 30342
I20260619 15:03:15.216516 30403 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 15:03:15.223161 30438 webserver.cc:533] Webserver: sslize failed:
E20260619 15:03:15.223299 30438 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 15:03:15.311465 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 30342
2026-06-19T15:03:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (751 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T15:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:03:15Z Disabled control of system clock
I20260619 15:03:15.395440 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43899
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:43899
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/testchainca.cert with env {}
W20260619 15:03:15.722447 30443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:15.722796 30443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:15.722941 30443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:15.731632 30443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:15.731753 30443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:15.731832 30443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:15.731918 30443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:15.731999 30443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 15:03:15.732084 30443 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 15:03:15.732156 30443 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 15:03:15.742868 30443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:43899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:15.744302 30443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:15.745687 30443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:15.758065 30448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:15.758111 30449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:15.758458 30451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:15.759048 30443 server_base.cc:1034] running on GCE node
I20260619 15:03:15.760064 30443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:15.761536 30443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:15.762754 30443 hybrid_clock.cc:648] HybridClock initialized: now 1781881395762700 us; error 43 us; skew 500 ppm
I20260619 15:03:15.763228 30443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:03:15.765697 30443 webserver.cc:460] Webserver started at http://127.24.103.126:38929/ using document root <none> and password file <none>
I20260619 15:03:15.766376 30443 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:15.766479 30443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:15.766728 30443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:15.769727 30443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "872bd48b651646f4993cb5af858ef921"
format_stamp: "Formatted at 2026-06-19 15:03:15 on dist-test-slave-zpbw"
server_key: "7d3adfc481b025a3c7c4ebddb3160e0f"
server_key_iv: "567608ed23026d90051982c666f7b92c"
server_key_version: "encryptionkey@0"
I20260619 15:03:15.770507 30443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "872bd48b651646f4993cb5af858ef921"
format_stamp: "Formatted at 2026-06-19 15:03:15 on dist-test-slave-zpbw"
server_key: "7d3adfc481b025a3c7c4ebddb3160e0f"
server_key_iv: "567608ed23026d90051982c666f7b92c"
server_key_version: "encryptionkey@0"
I20260619 15:03:15.775852 30443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 15:03:15.779398 30457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:15.780890 30443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:03:15.781080 30443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "872bd48b651646f4993cb5af858ef921"
format_stamp: "Formatted at 2026-06-19 15:03:15 on dist-test-slave-zpbw"
server_key: "7d3adfc481b025a3c7c4ebddb3160e0f"
server_key_iv: "567608ed23026d90051982c666f7b92c"
server_key_version: "encryptionkey@0"
I20260619 15:03:15.781308 30443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:15.809257 30443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:15.810158 30443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:15.810468 30443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:15.829048 30443 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:43899
I20260619 15:03:15.829097 30508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:43899 every 8 connection(s)
I20260619 15:03:15.830509 30443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:15.834528 30509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:15.836746 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 30443
I20260619 15:03:15.836930 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:03:15.837661 24989 external_mini_cluster.cc:1428] Setting key 5710f5eeab9a0f89edeec1f7993c2425
I20260619 15:03:15.846284 30509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921: Bootstrap starting.
I20260619 15:03:15.849440 30509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:15.850591 30509 log.cc:824] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:15.851161 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:15.839740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.24.103.126:43899)
0619 15:03:15.840792 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:15.840804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:15.840844 (+ 40us) server_negotiation.cc:407] Connection header received
0619 15:03:15.842485 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:15.842508 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:15.843032 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:15.843444 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:15.844021 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:15.845341 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:15.846650 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:15.847360 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:15.847571 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:15.848797 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:15.848804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:15.848817 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:03:15.849327 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:15.849433 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:03:15.849452 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:15.849575 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:15.849800 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 15:03:15.850115 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":374,"threads_started":1}
I20260619 15:03:15.853910 30509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921: No bootstrap required, opened a new log
I20260619 15:03:15.855293 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:03:15.861649 30509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872bd48b651646f4993cb5af858ef921" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43899 } }
I20260619 15:03:15.862080 30509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:15.862251 30509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 872bd48b651646f4993cb5af858ef921, State: Initialized, Role: FOLLOWER
I20260619 15:03:15.862967 30509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872bd48b651646f4993cb5af858ef921" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43899 } }
I20260619 15:03:15.863193 30509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:15.863291 30509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:15.863408 30509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:15.865464 30509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872bd48b651646f4993cb5af858ef921" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43899 } }
I20260619 15:03:15.865908 30509 leader_election.cc:302] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 872bd48b651646f4993cb5af858ef921; no voters:
I20260619 15:03:15.866484 30509 leader_election.cc:288] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:03:15.866638 30514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:15.868574 30509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:15.868925 30514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [term 1 LEADER]: Becoming Leader. State: Replica: 872bd48b651646f4993cb5af858ef921, State: Running, Role: LEADER
I20260619 15:03:15.869454 30514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872bd48b651646f4993cb5af858ef921" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43899 } }
I20260619 15:03:15.872701 30515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "872bd48b651646f4993cb5af858ef921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872bd48b651646f4993cb5af858ef921" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43899 } } }
I20260619 15:03:15.873051 30515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:15.872836 30516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 872bd48b651646f4993cb5af858ef921. Latest consensus state: current_term: 1 leader_uuid: "872bd48b651646f4993cb5af858ef921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872bd48b651646f4993cb5af858ef921" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43899 } } }
I20260619 15:03:15.873251 30516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921 [sys.catalog]: This master's current role is: LEADER
W20260619 15:03:15.880630 30529 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:03:15.880780 30529 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:03:15.880961 30525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:15.883100 30525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:15.892752 30525 catalog_manager.cc:1347] Generated new cluster ID: cd48f00a48394a22b193eb1a2b1f4513
I20260619 15:03:15.892891 30525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:15.905542 30525 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:03:15.907485 30525 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:15.920679 30525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 872bd48b651646f4993cb5af858ef921: Generated new TSK 0
I20260619 15:03:15.921842 30525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:03:15.984725 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:15.944032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.24.103.126:43899)
0619 15:03:15.944291 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:15.944301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:15.944781 (+ 480us) server_negotiation.cc:407] Connection header received
0619 15:03:15.945035 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:15.945039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:15.945128 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:15.945268 (+ 140us) server_negotiation.cc:226] Negotiated authn=JWT
0619 15:03:15.945821 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:15.946604 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:15.948747 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:15.949172 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:15.949270 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:15.949834 (+ 564us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 15:03:15.983978 (+ 34144us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 15:03:15.984387 (+ 409us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51114: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":86}
W20260619 15:03:15.984874 30512 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51114: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 15:03:15.984906 30538 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:15.943994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:43899 (local address 127.0.0.1:51114)
0619 15:03:15.944664 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:15.944684 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:15.944844 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:15.945318 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:15.945323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:15.945337 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0619 15:03:15.945651 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:15.945663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:15.946753 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:15.946759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:15.948554 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:15.948571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:15.949348 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:15.949355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:15.949536 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:15.949583 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 15:03:15.984305 (+ 34722us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 15:03:15.984552 (+ 247us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.103.126:43899: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":340,"threads_started":1}
I20260619 15:03:15.986014 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 30443
I20260619 15:03:15.987365 30504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:16.084282 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 30443
2026-06-19T15:03:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (751 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (414 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (418 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:03:17 dist-test-slave-zpbw krb5kdc[30553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:03:17 dist-test-slave-zpbw krb5kdc[30553](info): set up 2 sockets
Jun 19 15:03:17 dist-test-slave-zpbw krb5kdc[30553](info): commencing operation
krb5kdc: starting...
W20260619 15:03:17.999882 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:03:18 dist-test-slave-zpbw krb5kdc[30553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:03:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:03:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:03:18.211978 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:40989
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:40989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:18.533622 30566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:18.533916 30566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:18.534065 30566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:18.541474 30566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:18.541595 30566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:18.541672 30566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:18.541761 30566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:18.541841 30566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:03:18.552023 30566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:40989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:40989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:18.553443 30566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:18.554731 30566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:18.567147 30572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:18.567195 30574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:18.567142 30571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:18.567936 30566 server_base.cc:1034] running on GCE node
I20260619 15:03:18.568794 30566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:18.570269 30566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:18.571493 30566 hybrid_clock.cc:648] HybridClock initialized: now 1781881398571458 us; error 30 us; skew 500 ppm
I20260619 15:03:18.571854 30566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:18 dist-test-slave-zpbw krb5kdc[30553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881398, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:18.575067 30566 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:03:18.576646 30566 webserver.cc:460] Webserver started at http://127.24.103.126:43435/ using document root <none> and password file <none>
I20260619 15:03:18.577468 30566 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:18.577636 30566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:18.577973 30566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:18.581156 30566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "71c19200daa74a4ab338dc38d7202e22"
format_stamp: "Formatted at 2026-06-19 15:03:18 on dist-test-slave-zpbw"
server_key: "177ed662e7d4d55ffd8316561c6cf3c9"
server_key_iv: "cb72642757c22bc2e0d73df25857cd43"
server_key_version: "encryptionkey@0"
I20260619 15:03:18.582012 30566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "71c19200daa74a4ab338dc38d7202e22"
format_stamp: "Formatted at 2026-06-19 15:03:18 on dist-test-slave-zpbw"
server_key: "177ed662e7d4d55ffd8316561c6cf3c9"
server_key_iv: "cb72642757c22bc2e0d73df25857cd43"
server_key_version: "encryptionkey@0"
I20260619 15:03:18.587486 30566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 15:03:18.590996 30581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:18.592397 30566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:03:18.592607 30566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "71c19200daa74a4ab338dc38d7202e22"
format_stamp: "Formatted at 2026-06-19 15:03:18 on dist-test-slave-zpbw"
server_key: "177ed662e7d4d55ffd8316561c6cf3c9"
server_key_iv: "cb72642757c22bc2e0d73df25857cd43"
server_key_version: "encryptionkey@0"
I20260619 15:03:18.592825 30566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:18.614204 30566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:18.618710 30566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:18.619037 30566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:18.637894 30566 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:40989
I20260619 15:03:18.637921 30632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:40989 every 8 connection(s)
I20260619 15:03:18.639429 30566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:18.643235 30633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:18.643719 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 30566
I20260619 15:03:18.643937 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:03:18.644765 24989 external_mini_cluster.cc:1428] Setting key 3d54fc48cdfeff75d7a93c7c3646d9e3
I20260619 15:03:18.657887 30633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22: Bootstrap starting.
Jun 19 15:03:18 dist-test-slave-zpbw krb5kdc[30553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:18.661667 30633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:18.663093 30633 log.cc:824] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:18.666332 30633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22: No bootstrap required, opened a new log
I20260619 15:03:18.670024 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:18.646970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.24.103.126:40989)
0619 15:03:18.647808 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:18.647820 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:18.647869 (+ 49us) server_negotiation.cc:407] Connection header received
0619 15:03:18.649904 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:18.649933 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:18.650350 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:18.650740 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:18.652566 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:18.653884 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:18.655741 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:18.656449 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:18.656627 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:18.661455 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:18.661468 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:18.661490 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:18.661549 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:18.665391 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:18.666117 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:18.666125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:18.666132 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:18.666336 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:18.666772 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:18.666800 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:18.666804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:18.667506 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:18.667844 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:18.668225 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 15:03:18.668681 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":391,"threads_started":1}
I20260619 15:03:18.675488 30633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c19200daa74a4ab338dc38d7202e22" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40989 } }
I20260619 15:03:18.675839 30633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:18.675909 30633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c19200daa74a4ab338dc38d7202e22, State: Initialized, Role: FOLLOWER
I20260619 15:03:18.676388 30633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c19200daa74a4ab338dc38d7202e22" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40989 } }
I20260619 15:03:18.676599 30633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:18.676718 30633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:18.676847 30633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:18.679097 30633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c19200daa74a4ab338dc38d7202e22" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40989 } }
I20260619 15:03:18.679661 30633 leader_election.cc:302] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71c19200daa74a4ab338dc38d7202e22; no voters:
I20260619 15:03:18.680179 30633 leader_election.cc:288] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:03:18.680335 30638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:18.681087 30638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [term 1 LEADER]: Becoming Leader. State: Replica: 71c19200daa74a4ab338dc38d7202e22, State: Running, Role: LEADER
I20260619 15:03:18.681644 30638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c19200daa74a4ab338dc38d7202e22" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40989 } }
I20260619 15:03:18.682366 30633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:18.684667 30640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c19200daa74a4ab338dc38d7202e22. Latest consensus state: current_term: 1 leader_uuid: "71c19200daa74a4ab338dc38d7202e22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c19200daa74a4ab338dc38d7202e22" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40989 } } }
I20260619 15:03:18.685086 30640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:18.685572 30639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71c19200daa74a4ab338dc38d7202e22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c19200daa74a4ab338dc38d7202e22" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40989 } } }
I20260619 15:03:18.685868 30639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:18.687486 30646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:18.692790 30646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:18.702090 30646 catalog_manager.cc:1347] Generated new cluster ID: 4056258f9fb24ba095384b171f0950bf
I20260619 15:03:18.702219 30646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:18.728381 30646 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:03:18.729777 30646 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:18.736702 30646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71c19200daa74a4ab338dc38d7202e22: Generated new TSK 0
I20260619 15:03:18.737891 30646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:03:18.799652 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40989
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:19.161324 30659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:19.161821 30659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:19.162002 30659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:19.169785 30659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:19.169924 30659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:19.170044 30659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:03:19.180647 30659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:19.182121 30659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:19.183694 30659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:19.197913 30665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:19.198012 30664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:19.197971 30667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:19.198972 30659 server_base.cc:1034] running on GCE node
I20260619 15:03:19.199632 30659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:19.200621 30659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:19.201865 30659 hybrid_clock.cc:648] HybridClock initialized: now 1781881399201808 us; error 42 us; skew 500 ppm
I20260619 15:03:19.202241 30659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:19 dist-test-slave-zpbw krb5kdc[30553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:19.205719 30659 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:03:19.207332 30659 webserver.cc:460] Webserver started at http://127.24.103.65:46837/ using document root <none> and password file <none>
I20260619 15:03:19.208055 30659 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:19.208190 30659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:19.208544 30659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:19.211679 30659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "2d0395ec5a30480298438ca56ea0cfb9"
format_stamp: "Formatted at 2026-06-19 15:03:19 on dist-test-slave-zpbw"
server_key: "d14ea8149ff5093e6135b4215b20b58d"
server_key_iv: "e57991bdc2197c7ee4e47127e935d08c"
server_key_version: "encryptionkey@0"
I20260619 15:03:19.212471 30659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "2d0395ec5a30480298438ca56ea0cfb9"
format_stamp: "Formatted at 2026-06-19 15:03:19 on dist-test-slave-zpbw"
server_key: "d14ea8149ff5093e6135b4215b20b58d"
server_key_iv: "e57991bdc2197c7ee4e47127e935d08c"
server_key_version: "encryptionkey@0"
I20260619 15:03:19.217815 30659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:03:19.221464 30674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:19.222937 30659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:03:19.223120 30659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "2d0395ec5a30480298438ca56ea0cfb9"
format_stamp: "Formatted at 2026-06-19 15:03:19 on dist-test-slave-zpbw"
server_key: "d14ea8149ff5093e6135b4215b20b58d"
server_key_iv: "e57991bdc2197c7ee4e47127e935d08c"
server_key_version: "encryptionkey@0"
I20260619 15:03:19.223313 30659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:19.245808 30659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:19.250550 30659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:19.250871 30659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:19.251835 30659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:19.253194 30659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:19.253273 30659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:19.253356 30659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:19.253399 30659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:19.287055 30659 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:45627
I20260619 15:03:19.287091 30786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:45627 every 8 connection(s)
I20260619 15:03:19.288475 30659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:03:19.294224 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 30659
I20260619 15:03:19.294432 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:03:19.295195 24989 external_mini_cluster.cc:1428] Setting key fb64823eb5df23144b1f9e0b710a9fa7
Jun 19 15:03:19 dist-test-slave-zpbw krb5kdc[30553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:19.311431 30789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:19.291279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:48067 (local address 127.24.103.126:40989)
0619 15:03:19.291747 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:19.291751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:19.292669 (+ 918us) server_negotiation.cc:407] Connection header received
0619 15:03:19.294598 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:19.294604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:19.294707 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:19.294848 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:19.296741 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:19.297636 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:19.299343 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:19.299751 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:19.299893 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:19.303942 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:19.303952 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:19.303960 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:19.304008 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:19.307779 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:19.308669 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:19.308677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:19.308684 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:19.308844 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:19.309332 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:19.309340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:19.309346 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:19.309732 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:19.309887 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:19.310625 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 15:03:19.310937 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":260,"threads_started":1}
I20260619 15:03:19.312172 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:19.291749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:40989 (local address 127.24.103.65:48067)
0619 15:03:19.292499 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:19.292538 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:19.294240 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:19.295089 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:19.295097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:19.295792 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:19.296506 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:19.296532 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:19.297825 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:19.297830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:19.299196 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:19.299208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:19.299989 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:19.299997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:19.300250 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:19.300970 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:19.300995 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:19.303658 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:19.307969 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:19.307977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:19.307993 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:19.308519 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:19.308989 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:19.308994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:19.308996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:19.309206 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:19.309858 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:19.309864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:19.310219 (+ 355us) client_negotiation.cc:769] Sending connection context
0619 15:03:19.310497 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 15:03:19.311002 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":334,"threads_started":1}
I20260619 15:03:19.313875 30787 heartbeater.cc:347] Connected to a master server at 127.24.103.126:40989
I20260619 15:03:19.314204 30787 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:19.315017 30787 heartbeater.cc:511] Master 127.24.103.126:40989 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260619 15:03:19.317305 30598 ts_manager.cc:194] Registered new tserver with Master: 2d0395ec5a30480298438ca56ea0cfb9 (127.24.103.65:45627)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
I20260619 15:03:19.318992 30598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:48067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:03:19.342442 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40989
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:19.680655 30793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:19.681063 30793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:19.681227 30793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:19.695286 30793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:19.695472 30793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:19.695725 30793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:03:19.708611 30793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:19.709887 30793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:19.711242 30793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:19.724519 30799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:19.724502 30798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:19.724457 30801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:19.725525 30793 server_base.cc:1034] running on GCE node
I20260619 15:03:19.726120 30793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:19.727180 30793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:19.728499 30793 hybrid_clock.cc:648] HybridClock initialized: now 1781881399728395 us; error 51 us; skew 500 ppm
I20260619 15:03:19.728807 30793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:19 dist-test-slave-zpbw krb5kdc[30553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:19.732025 30793 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:03:19.733446 30793 webserver.cc:460] Webserver started at http://127.24.103.66:32947/ using document root <none> and password file <none>
I20260619 15:03:19.734078 30793 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:19.734185 30793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:19.734424 30793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:19.737481 30793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "435601dc5b5f40a68cda8edce53eb01b"
format_stamp: "Formatted at 2026-06-19 15:03:19 on dist-test-slave-zpbw"
server_key: "9fba7d4cefcb20913f785a14ecb7de7b"
server_key_iv: "3ff7d0e274de2410b379d95ea955c090"
server_key_version: "encryptionkey@0"
I20260619 15:03:19.738206 30793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "435601dc5b5f40a68cda8edce53eb01b"
format_stamp: "Formatted at 2026-06-19 15:03:19 on dist-test-slave-zpbw"
server_key: "9fba7d4cefcb20913f785a14ecb7de7b"
server_key_iv: "3ff7d0e274de2410b379d95ea955c090"
server_key_version: "encryptionkey@0"
I20260619 15:03:19.743677 30793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:03:19.748178 30808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:19.749629 30793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:03:19.749775 30793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "435601dc5b5f40a68cda8edce53eb01b"
format_stamp: "Formatted at 2026-06-19 15:03:19 on dist-test-slave-zpbw"
server_key: "9fba7d4cefcb20913f785a14ecb7de7b"
server_key_iv: "3ff7d0e274de2410b379d95ea955c090"
server_key_version: "encryptionkey@0"
I20260619 15:03:19.749951 30793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:19.770612 30793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:19.775121 30793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:19.775401 30793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:19.776296 30793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:19.777549 30793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:19.777622 30793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:19.777702 30793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:19.777745 30793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:19.811352 30793 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:42807
I20260619 15:03:19.811431 30920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:42807 every 8 connection(s)
I20260619 15:03:19.812809 30793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:03:19.813764 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 30793
I20260619 15:03:19.813962 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:03:19.814507 24989 external_mini_cluster.cc:1428] Setting key b5905766c5e10abb1552703ec69df451
Jun 19 15:03:19 dist-test-slave-zpbw krb5kdc[30553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260619 15:03:19.834517 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:19.815656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:51861 (local address 127.24.103.126:40989)
0619 15:03:19.816139 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:19.816145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:19.817279 (+ 1134us) server_negotiation.cc:407] Connection header received
0619 15:03:19.819121 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:19.819126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:19.819210 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:19.819338 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:19.820808 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:19.821539 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:19.823320 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:19.823752 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:19.823854 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:19.828333 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:19.828361 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:19.828368 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:19.828414 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:19.831271 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:19.832057 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:19.832062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:19.832064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:19.832193 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:19.832735 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:19.832741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:19.832744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:19.833020 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:19.833176 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:19.833916 (+ 740us) server_negotiation.cc:299] Negotiation successful
0619 15:03:19.834138 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":242,"threads_started":1}
I20260619 15:03:19.835680 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:19.816360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:40989 (local address 127.24.103.66:51861)
0619 15:03:19.817141 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:19.817177 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:19.818855 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:19.819576 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:19.819585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:19.820082 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:19.820638 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:19.820654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:19.821684 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:19.821696 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:19.823174 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:19.823191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:19.823981 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:19.823990 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:19.824332 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:19.825105 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:19.825145 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:19.828034 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:19.831428 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:19.831439 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:19.831460 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:19.831910 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:19.832309 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:19.832314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:19.832318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:19.832601 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:19.833154 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:19.833163 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:19.833555 (+ 392us) client_negotiation.cc:769] Sending connection context
0619 15:03:19.833827 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 15:03:19.834386 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":340,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
I20260619 15:03:19.837421 30921 heartbeater.cc:347] Connected to a master server at 127.24.103.126:40989
I20260619 15:03:19.837781 30921 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:19.838626 30921 heartbeater.cc:511] Master 127.24.103.126:40989 requested a full tablet report, sending...
I20260619 15:03:19.840353 30598 ts_manager.cc:194] Registered new tserver with Master: 435601dc5b5f40a68cda8edce53eb01b (127.24.103.66:42807)
I20260619 15:03:19.841635 30598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:51861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:03:19.862108 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40989
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:20.188277 30927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:20.188665 30927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:20.188836 30927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:20.196568 30927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:20.196719 30927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:20.196902 30927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:03:20.207059 30927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:20.208407 30927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:20.209755 30927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:20.222628 30933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:20.222632 30932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:20.223392 30935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:20.224071 30927 server_base.cc:1034] running on GCE node
I20260619 15:03:20.224718 30927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:20.225703 30927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:20.226903 30927 hybrid_clock.cc:648] HybridClock initialized: now 1781881400226861 us; error 31 us; skew 500 ppm
I20260619 15:03:20.227240 30927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:20.235368 30927 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:03:20.237165 30927 webserver.cc:460] Webserver started at http://127.24.103.67:44549/ using document root <none> and password file <none>
I20260619 15:03:20.237957 30927 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:20.238121 30927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:20.238479 30927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:20.241818 30927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "73e80953d44a407492c3ee70f8518fa8"
format_stamp: "Formatted at 2026-06-19 15:03:20 on dist-test-slave-zpbw"
server_key: "a1be4741ae56f71fda51e4e06c6268b3"
server_key_iv: "dc8c82fcbc46d8651ee20fd25cc37049"
server_key_version: "encryptionkey@0"
I20260619 15:03:20.242725 30927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "73e80953d44a407492c3ee70f8518fa8"
format_stamp: "Formatted at 2026-06-19 15:03:20 on dist-test-slave-zpbw"
server_key: "a1be4741ae56f71fda51e4e06c6268b3"
server_key_iv: "dc8c82fcbc46d8651ee20fd25cc37049"
server_key_version: "encryptionkey@0"
I20260619 15:03:20.248633 30927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:03:20.252529 30942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:20.254065 30927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 15:03:20.254261 30927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "73e80953d44a407492c3ee70f8518fa8"
format_stamp: "Formatted at 2026-06-19 15:03:20 on dist-test-slave-zpbw"
server_key: "a1be4741ae56f71fda51e4e06c6268b3"
server_key_iv: "dc8c82fcbc46d8651ee20fd25cc37049"
server_key_version: "encryptionkey@0"
I20260619 15:03:20.254514 30927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:20.265779 30927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:20.269989 30927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:20.270306 30927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:20.271432 30927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:20.272802 30927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:20.272909 30927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:20.273063 30927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:20.273136 30927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:20.307439 30927 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:41301
I20260619 15:03:20.307477 31054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:41301 every 8 connection(s)
I20260619 15:03:20.308799 30927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:03:20.312255 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 30927
I20260619 15:03:20.312454 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:03:20.312942 24989 external_mini_cluster.cc:1428] Setting key 8b946d6b847cdd35f07bceca46484299
I20260619 15:03:20.322036 30787 heartbeater.cc:503] Master 127.24.103.126:40989 was elected leader, sending a full tablet report...
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:20.329147 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:20.311431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:53549 (local address 127.24.103.126:40989)
0619 15:03:20.311647 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:20.311653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:20.312867 (+ 1214us) server_negotiation.cc:407] Connection header received
0619 15:03:20.315007 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:20.315012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:20.315103 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:20.315207 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:20.316889 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.317620 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.319158 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.319548 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.319640 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:20.323227 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:20.323233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:20.323238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:20.323271 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:20.325824 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.326508 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.326512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.326515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.326668 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.327297 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.327306 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.327308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.327664 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:20.327823 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:20.328747 (+ 924us) server_negotiation.cc:299] Negotiation successful
0619 15:03:20.328920 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 15:03:20.330188 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:20.311896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:40989 (local address 127.24.103.67:53549)
0619 15:03:20.312685 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:20.312740 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:20.314711 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:20.315474 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:20.315485 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:20.316204 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:20.316747 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:20.316763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.317748 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.317751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:20.319040 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:20.319054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.319727 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.319732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:20.319916 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:20.320595 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:20.320619 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:20.323026 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:20.325944 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:20.325951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:20.325964 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:20.326367 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:20.326879 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:20.326884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:20.326889 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:20.327155 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:20.327830 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:20.327838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:20.328301 (+ 463us) client_negotiation.cc:769] Sending connection context
0619 15:03:20.328641 (+ 340us) client_negotiation.cc:240] Negotiation successful
0619 15:03:20.329040 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":347,"threads_started":1}
I20260619 15:03:20.331620 31055 heartbeater.cc:347] Connected to a master server at 127.24.103.126:40989
I20260619 15:03:20.331884 31055 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:20.332515 31055 heartbeater.cc:511] Master 127.24.103.126:40989 requested a full tablet report, sending...
I20260619 15:03:20.333853 30598 ts_manager.cc:194] Registered new tserver with Master: 73e80953d44a407492c3ee70f8518fa8 (127.24.103.67:41301)
I20260619 15:03:20.334700 30598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:53549
I20260619 15:03:20.340859 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/test-admin.keytab.
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:20.368011 24989 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 15:03:20.373246 31062 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:20.370645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.24.103.65:45627)
0619 15:03:20.371343 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:20.371350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:20.371375 (+ 25us) server_negotiation.cc:407] Connection header received
0619 15:03:20.371439 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:20.371445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:20.371654 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:20.371839 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:20.372788 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48866: BlockingRecv error: recv got EOF from 127.0.0.1:48866 (error 108)
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":357,"threads_started":1}
W20260619 15:03:20.373292 24989 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/krb5cc' of type FILE
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:20.374938 24989 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:20.376299 24989 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:03:20.391989 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:20.377856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.24.103.65:45627)
0619 15:03:20.378041 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:20.378047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:20.378221 (+ 174us) server_negotiation.cc:407] Connection header received
0619 15:03:20.378428 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:20.378432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:20.378498 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:20.378599 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:20.379681 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.380652 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.381992 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.382571 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.382708 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:20.386367 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:20.386379 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:20.386394 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:20.386431 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:20.389154 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.389799 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.389809 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.389816 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.389946 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.390329 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.390335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.390338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.390697 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:20.391035 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:20.391426 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 15:03:20.391689 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 15:03:20.397991 31062 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:20.396207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.24.103.65:45627)
0619 15:03:20.396401 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:20.396406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:20.396611 (+ 205us) server_negotiation.cc:407] Connection header received
0619 15:03:20.396817 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:20.396826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:20.396880 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:20.396980 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:20.397745 (+ 765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48892: BlockingRecv error: recv got EOF from 127.0.0.1:48892 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 15:03:20.398218 24989 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/krb5cc' of type FILE
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:20.399801 24989 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:20.401038 24989 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:03:20.415009 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:20.402257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.24.103.65:45627)
0619 15:03:20.402457 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:20.402464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:20.402810 (+ 346us) server_negotiation.cc:407] Connection header received
0619 15:03:20.403002 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:20.403007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:20.403075 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:20.403175 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:20.404129 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.404831 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.406022 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.406454 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.406590 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:20.410203 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:20.410208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:20.410211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:20.410243 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:20.412567 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.413196 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.413208 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.413211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.413329 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.413704 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.413709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.413712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.413979 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:20.414234 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:20.414508 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 15:03:20.414711 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 15:03:20.419344 31062 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:20.417642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.24.103.65:45627)
0619 15:03:20.417810 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:20.417814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:20.418154 (+ 340us) server_negotiation.cc:407] Connection header received
0619 15:03:20.418268 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:20.418272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:20.418337 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:20.418412 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:20.419080 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48920: BlockingRecv error: recv got EOF from 127.0.0.1:48920 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 15:03:20.419552 24989 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881351219607-24989-0/krb5kdc/krb5cc' of type FILE
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:20.421082 24989 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:20.422343 24989 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:03:20.437701 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:20.423907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.24.103.65:45627)
0619 15:03:20.424123 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:20.424129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:20.424356 (+ 227us) server_negotiation.cc:407] Connection header received
0619 15:03:20.424569 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:20.424581 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:20.424639 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:20.424740 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:20.425704 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.426419 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.427879 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.428349 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.428494 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:20.432373 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:20.432379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:20.432382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:20.432418 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:20.435009 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.435720 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.435726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.435729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.435884 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.436264 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.436270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.436274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.436548 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:20.436869 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:20.437143 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 15:03:20.437367 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 15:03:20.439376 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 30659
I20260619 15:03:20.451750 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:20.440095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.24.103.65:45627)
0619 15:03:20.440255 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:20.440259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:20.440626 (+ 367us) server_negotiation.cc:407] Connection header received
0619 15:03:20.440746 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:20.440750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:20.440804 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:20.440881 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:20.441813 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.442490 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.443735 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.444166 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.444263 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:20.446536 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:20.446540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:20.446543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:20.446574 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:20.449123 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.449764 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.449769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.449771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.449943 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.450294 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.450298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.450300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.450620 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:20.450923 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:20.451308 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 15:03:20.451490 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 15:03:20.452314 30782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:20.536018 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 30659
I20260619 15:03:20.551973 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 30793
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:03:20.567348 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:20.552807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.24.103.66:42807)
0619 15:03:20.553389 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:20.553395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:20.553421 (+ 26us) server_negotiation.cc:407] Connection header received
0619 15:03:20.553479 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:20.553484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:20.553698 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:20.553822 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:20.554960 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.555969 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.557140 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.557742 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.557833 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:20.561821 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:20.561832 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:20.561847 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:20.561886 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:20.564763 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.565478 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.565484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.565487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.565607 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.565951 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.565955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.565956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.566280 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:20.566495 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:20.566872 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 15:03:20.567062 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":334,"threads_started":1}
I20260619 15:03:20.568638 30916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:20.658517 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 30793
I20260619 15:03:20.674048 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 30927
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[30553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:03:20.690212 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:20.675228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.24.103.67:41301)
0619 15:03:20.675894 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:20.675901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:20.675927 (+ 26us) server_negotiation.cc:407] Connection header received
0619 15:03:20.675985 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:20.675990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:20.676202 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:20.676324 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:20.677391 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.678355 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.679706 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:20.680242 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:20.680340 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:20.684339 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:20.684351 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:20.684366 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:20.684398 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:20.687424 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.688128 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.688135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.688137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.688250 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:20.688709 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:20.688712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:20.688714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:20.689042 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:20.689276 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:20.689576 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 15:03:20.689825 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":366,"threads_started":1}
I20260619 15:03:20.691897 31050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:20.775821 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 30927
I20260619 15:03:20.791282 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 30566
I20260619 15:03:20.792497 30628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:20.897996 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 30566
2026-06-19T15:03:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3987 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[31082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[31082](info): set up 2 sockets
Jun 19 15:03:20 dist-test-slave-zpbw krb5kdc[31082](info): commencing operation
krb5kdc: starting...
W20260619 15:03:21.995790 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.006s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:03:22 dist-test-slave-zpbw krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:03:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:03:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.126.keytab.
Entry for principal oryx/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.126.keytab.
I20260619 15:03:22.203703 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:36187
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.126.keytab
--principal=oryx/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:36187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:22.528142 31095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:22.528579 31095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:22.528744 31095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:22.536142 31095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:22.536304 31095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:22.536412 31095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:22.536509 31095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:22.536582 31095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:03:22.546595 31095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:36187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.126.keytab
--principal=oryx/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:36187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:22.548066 31095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:22.549407 31095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:22.561313 31101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:22.561440 31100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:22.561499 31103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:22.562650 31095 server_base.cc:1034] running on GCE node
I20260619 15:03:22.563531 31095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:22.564965 31095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:22.566200 31095 hybrid_clock.cc:648] HybridClock initialized: now 1781881402566143 us; error 56 us; skew 500 ppm
I20260619 15:03:22.566577 31095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:22 dist-test-slave-zpbw krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881402, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:22.569872 31095 init.cc:377] Logged in from keytab as oryx/127.24.103.126@KRBTEST.COM (short username oryx)
I20260619 15:03:22.571401 31095 webserver.cc:460] Webserver started at http://127.24.103.126:40369/ using document root <none> and password file <none>
I20260619 15:03:22.572042 31095 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:22.572144 31095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:22.572459 31095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:22.575433 31095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "0e27d27671e74cb798fe7148e4eb0ce8"
format_stamp: "Formatted at 2026-06-19 15:03:22 on dist-test-slave-zpbw"
server_key: "6f497f4390b1ffebe424d67d3b4da297"
server_key_iv: "b2ed3a240ee400735184f41b54f16616"
server_key_version: "encryptionkey@0"
I20260619 15:03:22.576136 31095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "0e27d27671e74cb798fe7148e4eb0ce8"
format_stamp: "Formatted at 2026-06-19 15:03:22 on dist-test-slave-zpbw"
server_key: "6f497f4390b1ffebe424d67d3b4da297"
server_key_iv: "b2ed3a240ee400735184f41b54f16616"
server_key_version: "encryptionkey@0"
I20260619 15:03:22.581215 31095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:03:22.584621 31110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:22.585918 31095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 15:03:22.586052 31095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "0e27d27671e74cb798fe7148e4eb0ce8"
format_stamp: "Formatted at 2026-06-19 15:03:22 on dist-test-slave-zpbw"
server_key: "6f497f4390b1ffebe424d67d3b4da297"
server_key_iv: "b2ed3a240ee400735184f41b54f16616"
server_key_version: "encryptionkey@0"
I20260619 15:03:22.586221 31095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:22.602234 31095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:22.606217 31095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:22.606457 31095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:22.624632 31095 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:36187
I20260619 15:03:22.624655 31161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:36187 every 8 connection(s)
I20260619 15:03:22.626041 31095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:22.629741 31162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:22.633888 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 31095
I20260619 15:03:22.634111 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:03:22.634938 24989 external_mini_cluster.cc:1428] Setting key 45635569ba9bd5c1ce0efc57116788bd
I20260619 15:03:22.643740 31162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8: Bootstrap starting.
I20260619 15:03:22.647511 31162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:22.648842 31162 log.cc:824] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8: Log is configured to *not* fsync() on all Append() calls
Jun 19 15:03:22 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:22.651906 31162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8: No bootstrap required, opened a new log
I20260619 15:03:22.658113 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:22.637124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58712 (local address 127.24.103.126:36187)
0619 15:03:22.637903 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:22.637914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:22.637954 (+ 40us) server_negotiation.cc:407] Connection header received
0619 15:03:22.639584 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:22.639615 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:22.640022 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:22.640347 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:22.642128 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:22.643483 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:22.645202 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:22.645884 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:22.646011 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:22.650959 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:22.650966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:22.650980 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:22.651014 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:22.654111 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:22.654937 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:22.654943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:22.654949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:22.655104 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:22.655518 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:22.655521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:22.655523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:22.656000 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:22.656246 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:22.656671 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 15:03:22.657071 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":366,"threads_started":1}
I20260619 15:03:22.660966 31162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } }
I20260619 15:03:22.661394 31162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:22.661499 31162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e27d27671e74cb798fe7148e4eb0ce8, State: Initialized, Role: FOLLOWER
I20260619 15:03:22.662032 31162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } }
I20260619 15:03:22.662318 31162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:22.662489 31162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:22.662650 31162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:22.665020 31162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } }
I20260619 15:03:22.665524 31162 leader_election.cc:302] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e27d27671e74cb798fe7148e4eb0ce8; no voters:
I20260619 15:03:22.666133 31162 leader_election.cc:288] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:03:22.666273 31167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:22.667520 31167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 1 LEADER]: Becoming Leader. State: Replica: 0e27d27671e74cb798fe7148e4eb0ce8, State: Running, Role: LEADER
I20260619 15:03:22.668040 31167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } }
I20260619 15:03:22.668251 31162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:22.671188 31168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } }
I20260619 15:03:22.671157 31169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e27d27671e74cb798fe7148e4eb0ce8. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } }
I20260619 15:03:22.671589 31168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:22.671635 31169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:22.678433 31174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:22.683884 31174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:22.692170 31174 catalog_manager.cc:1347] Generated new cluster ID: b9aec6e2a07b498ba36c28cb897b5765
I20260619 15:03:22.692289 31174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:22.706126 31174 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:03:22.707505 31174 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:22.712756 31174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8: Generated new TSK 0
I20260619 15:03:22.713718 31174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab.
Entry for principal oryx/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab.
I20260619 15:03:22.769999 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36187
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:23.102036 31188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:23.102345 31188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:23.102496 31188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:23.109952 31188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:23.110071 31188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:23.110283 31188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:03:23.120153 31188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:23.121503 31188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:23.122885 31188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:23.136576 31193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:23.136672 31194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:23.136699 31196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:23.137562 31188 server_base.cc:1034] running on GCE node
I20260619 15:03:23.138283 31188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:23.139384 31188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:23.140594 31188 hybrid_clock.cc:648] HybridClock initialized: now 1781881403140544 us; error 40 us; skew 500 ppm
I20260619 15:03:23.140920 31188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:23 dist-test-slave-zpbw krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881403, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:23.144200 31188 init.cc:377] Logged in from keytab as oryx/127.24.103.65@KRBTEST.COM (short username oryx)
I20260619 15:03:23.145763 31188 webserver.cc:460] Webserver started at http://127.24.103.65:37031/ using document root <none> and password file <none>
I20260619 15:03:23.146481 31188 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:23.146633 31188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:23.147004 31188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:23.150038 31188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "76c90e3825b54f8f91d82c5ba5e9c102"
format_stamp: "Formatted at 2026-06-19 15:03:23 on dist-test-slave-zpbw"
server_key: "85c604d27c0b63ee49d1e5497b68355f"
server_key_iv: "fbdf1395631d94967b19a3156ccd6413"
server_key_version: "encryptionkey@0"
I20260619 15:03:23.150913 31188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "76c90e3825b54f8f91d82c5ba5e9c102"
format_stamp: "Formatted at 2026-06-19 15:03:23 on dist-test-slave-zpbw"
server_key: "85c604d27c0b63ee49d1e5497b68355f"
server_key_iv: "fbdf1395631d94967b19a3156ccd6413"
server_key_version: "encryptionkey@0"
I20260619 15:03:23.156325 31188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 15:03:23.160102 31203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:23.161497 31188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:03:23.161680 31188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "76c90e3825b54f8f91d82c5ba5e9c102"
format_stamp: "Formatted at 2026-06-19 15:03:23 on dist-test-slave-zpbw"
server_key: "85c604d27c0b63ee49d1e5497b68355f"
server_key_iv: "fbdf1395631d94967b19a3156ccd6413"
server_key_version: "encryptionkey@0"
I20260619 15:03:23.161901 31188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:23.178699 31188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:23.182726 31188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:23.183132 31188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:23.184080 31188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:23.185592 31188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:23.185699 31188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:23.185863 31188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:23.185932 31188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:23.219460 31188 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:38491
I20260619 15:03:23.219511 31315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:38491 every 8 connection(s)
I20260619 15:03:23.220798 31188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:03:23.226581 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 31188
I20260619 15:03:23.226854 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:03:23.227524 24989 external_mini_cluster.cc:1428] Setting key afec2ef8562149c463fbcf6351421f75
Jun 19 15:03:23 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881403, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:23.241389 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:23.223434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:54637 (local address 127.24.103.126:36187)
0619 15:03:23.223849 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:23.223854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:23.224897 (+ 1043us) server_negotiation.cc:407] Connection header received
0619 15:03:23.226682 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:23.226688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:23.226831 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:23.226954 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:23.228536 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:23.229341 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:23.230961 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:23.231345 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:23.231482 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:23.235272 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:23.235278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:23.235281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:23.235315 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:23.237962 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:23.238994 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:23.238998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:23.239000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:23.239124 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:23.239665 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:23.239671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:23.239673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:23.239948 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:23.240070 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:23.240965 (+ 895us) server_negotiation.cc:299] Negotiation successful
0619 15:03:23.241142 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":212,"threads_started":1}
I20260619 15:03:23.243042 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:23.224044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.65:54637)
0619 15:03:23.224737 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:23.224780 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:23.226402 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:23.227208 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:23.227217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:23.227806 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:23.228366 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:23.228383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:23.229528 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:23.229532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:23.230750 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:23.230762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:23.231574 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:23.231581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:23.231819 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:23.232534 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:23.232561 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:23.234995 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:23.238132 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:23.238145 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:23.238166 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:23.238874 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:23.239239 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:23.239247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:23.239253 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:23.239515 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:23.240074 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:23.240086 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:23.240560 (+ 474us) client_negotiation.cc:769] Sending connection context
0619 15:03:23.240886 (+ 326us) client_negotiation.cc:240] Negotiation successful
0619 15:03:23.241494 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":259,"threads_started":1}
I20260619 15:03:23.244746 31316 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36187
I20260619 15:03:23.245086 31316 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:23.245867 31316 heartbeater.cc:511] Master 127.24.103.126:36187 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260619 15:03:23.248129 31127 ts_manager.cc:194] Registered new tserver with Master: 76c90e3825b54f8f91d82c5ba5e9c102 (127.24.103.65:38491)
I20260619 15:03:23.249610 31127 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:54637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab.
Entry for principal oryx/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab.
I20260619 15:03:23.274255 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36187
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:23.604702 31322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:23.604996 31322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:23.605201 31322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:23.612920 31322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:23.613024 31322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:23.613214 31322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:03:23.622848 31322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:23.624308 31322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:23.625617 31322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:23.638520 31328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:23.638520 31327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:23.639309 31330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:23.639585 31322 server_base.cc:1034] running on GCE node
I20260619 15:03:23.640225 31322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:23.641216 31322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:23.642457 31322 hybrid_clock.cc:648] HybridClock initialized: now 1781881403642397 us; error 59 us; skew 500 ppm
I20260619 15:03:23.642841 31322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:23 dist-test-slave-zpbw krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881403, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:23.645980 31322 init.cc:377] Logged in from keytab as oryx/127.24.103.66@KRBTEST.COM (short username oryx)
I20260619 15:03:23.647539 31322 webserver.cc:460] Webserver started at http://127.24.103.66:41097/ using document root <none> and password file <none>
I20260619 15:03:23.648269 31322 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:23.648425 31322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:23.648743 31322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:23.651726 31322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "638989ea9b4c4df599319b3f97e1b5d9"
format_stamp: "Formatted at 2026-06-19 15:03:23 on dist-test-slave-zpbw"
server_key: "cf78cb434b672d3cfbf3be5e522eaeb4"
server_key_iv: "30452809cdb5446ee52e5984f6934e63"
server_key_version: "encryptionkey@0"
I20260619 15:03:23.652483 31322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "638989ea9b4c4df599319b3f97e1b5d9"
format_stamp: "Formatted at 2026-06-19 15:03:23 on dist-test-slave-zpbw"
server_key: "cf78cb434b672d3cfbf3be5e522eaeb4"
server_key_iv: "30452809cdb5446ee52e5984f6934e63"
server_key_version: "encryptionkey@0"
I20260619 15:03:23.657612 31322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:03:23.660939 31337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:23.662204 31322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 15:03:23.662410 31322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "638989ea9b4c4df599319b3f97e1b5d9"
format_stamp: "Formatted at 2026-06-19 15:03:23 on dist-test-slave-zpbw"
server_key: "cf78cb434b672d3cfbf3be5e522eaeb4"
server_key_iv: "30452809cdb5446ee52e5984f6934e63"
server_key_version: "encryptionkey@0"
I20260619 15:03:23.662600 31322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:23.692785 31322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:23.696775 31322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:23.697096 31322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:23.698037 31322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:23.699479 31322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:23.699582 31322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:23.699735 31322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:23.699816 31322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:23.732431 31322 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:34037
I20260619 15:03:23.732477 31449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:34037 every 8 connection(s)
I20260619 15:03:23.733759 31322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:03:23.735870 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 31322
I20260619 15:03:23.736058 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:03:23.736636 24989 external_mini_cluster.cc:1428] Setting key e552e169614d0716d1d994747804849e
Jun 19 15:03:23 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881403, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:23.753831 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:23.736365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:33465 (local address 127.24.103.126:36187)
0619 15:03:23.736583 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:23.736598 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:23.737867 (+ 1269us) server_negotiation.cc:407] Connection header received
0619 15:03:23.739754 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:23.739760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:23.739851 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:23.739963 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:23.741441 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:23.742151 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:23.743917 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:23.744357 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:23.744490 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:23.748024 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:23.748029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:23.748033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:23.748065 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:23.750681 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:23.751482 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:23.751493 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:23.751495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:23.751622 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:23.752063 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:23.752073 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:23.752075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:23.752356 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:23.752528 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:23.753238 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 15:03:23.753474 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
WARNING: no policy specified for oryx/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260619 15:03:23.754940 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:23.736916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.66:33465)
0619 15:03:23.737700 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:23.737751 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:23.739481 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:23.740203 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:23.740212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:23.740696 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:23.741264 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:23.741281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:23.742304 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:23.742314 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:23.743745 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:23.743762 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:23.744592 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:23.744599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:23.744836 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:23.745522 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:23.745544 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:23.747793 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:23.750862 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:23.750871 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:23.750889 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:23.751350 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:23.751734 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:23.751739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:23.751743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:23.751940 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:23.752499 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:23.752507 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:23.752875 (+ 368us) client_negotiation.cc:769] Sending connection context
0619 15:03:23.753135 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 15:03:23.753693 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":349,"threads_started":1}
I20260619 15:03:23.756341 31450 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36187
I20260619 15:03:23.756651 31450 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.67@KRBTEST.COM" created.
I20260619 15:03:23.757483 31450 heartbeater.cc:511] Master 127.24.103.126:36187 requested a full tablet report, sending...
I20260619 15:03:23.759157 31127 ts_manager.cc:194] Registered new tserver with Master: 638989ea9b4c4df599319b3f97e1b5d9 (127.24.103.66:34037)
I20260619 15:03:23.760340 31127 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:33465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab.
Entry for principal oryx/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab.
I20260619 15:03:23.783905 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36187
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:24.110843 31455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:24.111315 31455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:24.111439 31455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:24.118716 31455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:24.118927 31455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:24.119062 31455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:03:24.128937 31455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:24.130208 31455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:24.131515 31455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:24.144831 31460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:24.144866 31463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:24.144889 31461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:24.146214 31455 server_base.cc:1034] running on GCE node
I20260619 15:03:24.146842 31455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:24.147804 31455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:24.149034 31455 hybrid_clock.cc:648] HybridClock initialized: now 1781881404148968 us; error 66 us; skew 500 ppm
I20260619 15:03:24.149369 31455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:24 dist-test-slave-zpbw krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881404, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:24.152691 31455 init.cc:377] Logged in from keytab as oryx/127.24.103.67@KRBTEST.COM (short username oryx)
I20260619 15:03:24.154184 31455 webserver.cc:460] Webserver started at http://127.24.103.67:42793/ using document root <none> and password file <none>
I20260619 15:03:24.154927 31455 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:24.155047 31455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:24.155325 31455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:24.158351 31455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "8f29062c40dd49a8952f4d893bec9fb5"
format_stamp: "Formatted at 2026-06-19 15:03:24 on dist-test-slave-zpbw"
server_key: "0ed15d5f2d833f16ae51be1f137b720b"
server_key_iv: "1a11c1c72e8abf4bd0ee19f4f5132d18"
server_key_version: "encryptionkey@0"
I20260619 15:03:24.159351 31455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "8f29062c40dd49a8952f4d893bec9fb5"
format_stamp: "Formatted at 2026-06-19 15:03:24 on dist-test-slave-zpbw"
server_key: "0ed15d5f2d833f16ae51be1f137b720b"
server_key_iv: "1a11c1c72e8abf4bd0ee19f4f5132d18"
server_key_version: "encryptionkey@0"
I20260619 15:03:24.164866 31455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 15:03:24.168883 31470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:24.170347 31455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:03:24.170478 31455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "8f29062c40dd49a8952f4d893bec9fb5"
format_stamp: "Formatted at 2026-06-19 15:03:24 on dist-test-slave-zpbw"
server_key: "0ed15d5f2d833f16ae51be1f137b720b"
server_key_iv: "1a11c1c72e8abf4bd0ee19f4f5132d18"
server_key_version: "encryptionkey@0"
I20260619 15:03:24.170658 31455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:24.189671 31455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:24.193809 31455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:24.194048 31455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:24.194985 31455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:24.196331 31455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:24.196403 31455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:24.196491 31455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:24.196533 31455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:24.230064 31455 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:46311
I20260619 15:03:24.230108 31582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:46311 every 8 connection(s)
I20260619 15:03:24.231503 31455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:03:24.234452 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 31455
I20260619 15:03:24.234659 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:03:24.235219 24989 external_mini_cluster.cc:1428] Setting key 24fb777507a9153c847b943539515821
Jun 19 15:03:24 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881404, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:24.252533 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.234173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:43239 (local address 127.24.103.126:36187)
0619 15:03:24.234404 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:24.234412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:24.235833 (+ 1421us) server_negotiation.cc:407] Connection header received
0619 15:03:24.238181 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:24.238185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:24.238280 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:24.238390 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:24.239965 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.240731 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.242309 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.242722 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.242926 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.246357 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:24.246362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:24.246365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:24.246401 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:24.248974 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.249933 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.249941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.249947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.250094 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.250691 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.250699 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.250702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.251046 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:24.251213 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:24.251937 (+ 724us) server_negotiation.cc:299] Negotiation successful
0619 15:03:24.252168 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 15:03:24.252753 31316 heartbeater.cc:503] Master 127.24.103.126:36187 was elected leader, sending a full tablet report...
I20260619 15:03:24.253659 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.234713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.67:43239)
0619 15:03:24.235640 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:24.235697 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:24.237879 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:24.238646 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:24.238654 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:24.239223 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:24.239796 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:24.239813 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.240889 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.240893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:24.242155 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:24.242167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.243015 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.243025 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:24.243275 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.243897 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:24.243928 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:24.246128 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:24.249184 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:24.249195 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:24.249213 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:24.249817 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:24.250385 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:24.250389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:24.250391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:24.250575 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:24.251185 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:24.251194 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:24.251569 (+ 375us) client_negotiation.cc:769] Sending connection context
0619 15:03:24.251856 (+ 287us) client_negotiation.cc:240] Negotiation successful
0619 15:03:24.252432 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":343,"threads_started":1}
I20260619 15:03:24.255200 31583 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36187
I20260619 15:03:24.255504 31583 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:24.256233 31583 heartbeater.cc:511] Master 127.24.103.126:36187 requested a full tablet report, sending...
I20260619 15:03:24.257642 31127 ts_manager.cc:194] Registered new tserver with Master: 8f29062c40dd49a8952f4d893bec9fb5 (127.24.103.67:46311)
I20260619 15:03:24.258450 31127 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:43239
I20260619 15:03:24.263331 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.103.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.125.keytab.
Entry for principal oryx/127.24.103.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.125.keytab.
I20260619 15:03:24.316375 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:44261
--webserver_interface=127.24.103.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.125.keytab
--principal=oryx/127.24.103.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:36187,127.24.103.125:44261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:24.647276 31588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:24.647565 31588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:24.647759 31588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:24.655328 31588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:24.655437 31588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:24.655536 31588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:24.655652 31588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:24.655738 31588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:03:24.665978 31588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:36187,127.24.103.125:44261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.125.keytab
--principal=oryx/127.24.103.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:44261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.103.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
W20260619 15:03:24.666437 31588 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.103.126:36187,127.24.103.125:44261'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 15:03:24.667675 31588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:24.669198 31588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:24.681989 31593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:24.681878 31596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:24.681906 31594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:24.683261 31588 server_base.cc:1034] running on GCE node
I20260619 15:03:24.684127 31588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:24.685640 31588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:24.686901 31588 hybrid_clock.cc:648] HybridClock initialized: now 1781881404686835 us; error 41 us; skew 500 ppm
I20260619 15:03:24.687260 31588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:24 dist-test-slave-zpbw krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881404, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:24.690516 31588 init.cc:377] Logged in from keytab as oryx/127.24.103.125@KRBTEST.COM (short username oryx)
I20260619 15:03:24.692061 31588 webserver.cc:460] Webserver started at http://127.24.103.125:38477/ using document root <none> and password file <none>
I20260619 15:03:24.692725 31588 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:24.692807 31588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:24.693063 31588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:24.696115 31588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data/instance:
uuid: "8fe5da68b9e34c32aa5758e08a0bdd53"
format_stamp: "Formatted at 2026-06-19 15:03:24 on dist-test-slave-zpbw"
server_key: "990f7dd31dead5dc2e47fa4670733a83"
server_key_iv: "5daf4f5ac75cac07e018e7356a7f52aa"
server_key_version: "encryptionkey@0"
I20260619 15:03:24.696810 31588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/wal/instance:
uuid: "8fe5da68b9e34c32aa5758e08a0bdd53"
format_stamp: "Formatted at 2026-06-19 15:03:24 on dist-test-slave-zpbw"
server_key: "990f7dd31dead5dc2e47fa4670733a83"
server_key_iv: "5daf4f5ac75cac07e018e7356a7f52aa"
server_key_version: "encryptionkey@0"
I20260619 15:03:24.702046 31588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 15:03:24.705502 31603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:24.706990 31588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:03:24.707181 31588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/wal
uuid: "8fe5da68b9e34c32aa5758e08a0bdd53"
format_stamp: "Formatted at 2026-06-19 15:03:24 on dist-test-slave-zpbw"
server_key: "990f7dd31dead5dc2e47fa4670733a83"
server_key_iv: "5daf4f5ac75cac07e018e7356a7f52aa"
server_key_version: "encryptionkey@0"
I20260619 15:03:24.707392 31588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:24.730084 31588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:24.734753 31588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:24.735085 31588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:24.753821 31588 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.125:44261
I20260619 15:03:24.753829 31654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.125:44261 every 8 connection(s)
I20260619 15:03:24.755313 31588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
I20260619 15:03:24.757642 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 31588
I20260619 15:03:24.759975 31655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:24.763398 31450 heartbeater.cc:503] Master 127.24.103.126:36187 was elected leader, sending a full tablet report...
I20260619 15:03:24.769532 31655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:03:24 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
Jun 19 15:03:24 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881404, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.125@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:24.784668 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.759894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.24.103.125:44261)
0619 15:03:24.760837 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:24.760850 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:24.760890 (+ 40us) server_negotiation.cc:407] Connection header received
0619 15:03:24.762945 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:24.762974 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:24.763489 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:24.763968 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:24.766307 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.768362 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.769834 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.770937 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.771136 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.776973 (+ 5837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:24.776981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:24.776990 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:24.777027 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:24.780173 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.780852 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.780860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.780869 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.781112 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.781518 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.781525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.781530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.782092 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:24.782438 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:24.782935 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 15:03:24.783379 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"spinlock_wait_cycles":16768,"thread_start_us":497,"threads_started":1}
I20260619 15:03:24.790393 31659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.772417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.24.103.126:36187)
0619 15:03:24.772925 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:24.772929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:24.775674 (+ 2745us) server_negotiation.cc:407] Connection header received
0619 15:03:24.775878 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:24.775883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:24.775972 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:24.776060 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:24.777481 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.778659 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.780124 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.780604 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.780733 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.784088 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:24.784095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:24.784099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:24.784150 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:24.787477 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.788316 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.788322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.788326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.788464 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.788873 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.788877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.788879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.789188 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:24.789346 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:24.789780 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 15:03:24.790018 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":276,"threads_started":1}
I20260619 15:03:24.790426 31660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.772454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:58714)
0619 15:03:24.775508 (+ 3054us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:24.775560 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:24.775736 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:24.776167 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:24.776175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:24.777027 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:24.777326 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:24.777335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.778844 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.778849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:24.779978 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:24.779988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.780791 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.780797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:24.780987 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.781634 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:24.781656 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:24.783888 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:24.787630 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:24.787640 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:24.787647 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:24.788182 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:24.788576 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:24.788582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:24.788586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:24.788729 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:24.789321 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:24.789329 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:24.789496 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 15:03:24.789679 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 15:03:24.790012 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2785,"thread_start_us":2532,"threads_started":1}
I20260619 15:03:24.792258 31655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:03:24 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881404, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.125@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
I20260619 15:03:24.803664 31660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.792984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:44261 (local address 127.0.0.1:50494)
0619 15:03:24.793203 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:24.793218 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:24.793296 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:24.793541 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:24.793544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:24.793845 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:24.793973 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:24.793981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.794936 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.794940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:24.795794 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:24.795804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.796408 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.796412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:24.796543 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.797064 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:24.797079 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:24.799071 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:24.801711 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:24.801715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:24.801717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:24.802047 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:24.802383 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:24.802386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:24.802388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:24.802504 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:24.803001 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:24.803004 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:24.803120 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 15:03:24.803225 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 15:03:24.803428 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 15:03:24.803993 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.793123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.24.103.125:44261)
0619 15:03:24.793270 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:24.793275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:24.793289 (+ 14us) server_negotiation.cc:407] Connection header received
0619 15:03:24.793383 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:24.793387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:24.793437 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:24.793509 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:24.794080 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.794760 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.795906 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.796292 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.796378 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.799234 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:24.799238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:24.799241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:24.799269 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:24.801588 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.802152 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.802156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.802158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.802294 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.802586 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.802589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.802591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.802889 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:24.802992 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:24.803306 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 15:03:24.803585 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":54}
I20260619 15:03:24.808806 31655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: Bootstrap starting.
I20260619 15:03:24.812067 31655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:24.813097 31655 log.cc:824] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:24.815799 31655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: No bootstrap required, opened a new log
I20260619 15:03:24.822937 31655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } }
I20260619 15:03:24.823307 31655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:24.823405 31655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe5da68b9e34c32aa5758e08a0bdd53, State: Initialized, Role: FOLLOWER
I20260619 15:03:24.824080 31655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } }
I20260619 15:03:24.824944 31662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } } }
I20260619 15:03:24.825286 31662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:03:24.826001 31655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 15:03:24.835119 31673 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:03:24.835309 31673 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:03:24.837608 31588 master_runner.cc:428] Detected that this master 8fe5da68b9e34c32aa5758e08a0bdd53 is joining an existing cluster
I20260619 15:03:24.837716 31588 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.103.125:44261
I20260619 15:03:24.858974 31659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.842235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.24.103.126:36187)
0619 15:03:24.842447 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:24.842456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:24.843266 (+ 810us) server_negotiation.cc:407] Connection header received
0619 15:03:24.843523 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:24.843530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:24.843652 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:24.843734 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:24.845262 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.846376 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.848474 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.849140 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.849241 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.851940 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:24.851947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:24.851952 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:24.851994 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:24.855682 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.856504 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.856510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.856514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.856689 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.857159 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.857164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.857168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.857556 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:24.857696 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:24.858319 (+ 623us) server_negotiation.cc:299] Negotiation successful
0619 15:03:24.858579 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 15:03:24.860452 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.842255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:58726)
0619 15:03:24.843158 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:24.843183 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:24.843375 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:24.843951 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:24.843956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:24.844673 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:24.845062 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:24.845075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.846576 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.846581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:24.848303 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:24.848322 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.849531 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.849537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:24.849714 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.850408 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:24.850431 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:24.851715 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:24.855845 (+ 4130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:24.855850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:24.855855 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:24.856365 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:24.856835 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:24.856840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:24.856844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:24.857009 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:24.857773 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:24.857779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:24.858005 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 15:03:24.859553 (+ 1548us) client_negotiation.cc:240] Negotiation successful
0619 15:03:24.859929 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":364,"threads_started":1}
WARNING: no policy specified for oryx/127.24.103.124@KRBTEST.COM; defaulting to no policy
I20260619 15:03:24.867004 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.843709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:44261 (local address 127.0.0.1:50510)
0619 15:03:24.850278 (+ 6569us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:24.850317 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:24.850642 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:24.851162 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:24.851169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:24.851930 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:24.852269 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:24.852284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.853719 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.853726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:24.855470 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:24.855487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.856810 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.856818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:24.857054 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.857665 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:24.857695 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:24.858925 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:24.864002 (+ 5077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:24.864009 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:24.864013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:24.864519 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:24.864933 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:24.864936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:24.864938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:24.865096 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:24.865724 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:24.865729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:24.865865 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 15:03:24.866073 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 15:03:24.866361 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6409,"thread_start_us":6691,"threads_started":1}
I20260619 15:03:24.866973 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.843420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.24.103.125:44261)
0619 15:03:24.843588 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:24.843596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:24.850395 (+ 6799us) server_negotiation.cc:407] Connection header received
0619 15:03:24.850794 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:24.850802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:24.850883 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:24.850971 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:24.852408 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.853518 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.855611 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.856396 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.856500 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.859126 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:24.859134 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:24.859141 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:24.859181 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:24.863834 (+ 4653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.864646 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.864651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.864654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.864765 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.865205 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.865209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.865212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.865613 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:24.865773 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:24.866227 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 15:03:24.866532 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.124@KRBTEST.COM" created.
Jun 19 15:03:24 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881402, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.126@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
I20260619 15:03:24.888228 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.867781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:44261 (local address 127.0.0.1:50518)
0619 15:03:24.868405 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:24.868447 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:24.868600 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:24.869349 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:24.869357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:24.870322 (+ 965us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:24.870624 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:24.870639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.872285 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.872290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:24.873906 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:24.873923 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.875372 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.875378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:24.875580 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.876391 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:24.876411 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:24.878980 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:24.884718 (+ 5738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:24.884736 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:24.884744 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:24.885464 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:24.886039 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:24.886044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:24.886047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:24.886256 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:24.887042 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:24.887052 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:24.887280 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 15:03:24.887506 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 15:03:24.887934 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":1162,"threads_started":1}
I20260619 15:03:24.889195 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:24.868776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50518 (local address 127.24.103.125:44261)
0619 15:03:24.868941 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:24.868949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:24.868966 (+ 17us) server_negotiation.cc:407] Connection header received
0619 15:03:24.869042 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:24.869046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:24.869147 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:24.869236 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:24.870862 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.872132 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.874050 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:24.874925 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:24.875153 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:24.879162 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:24.879170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:24.879179 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:24.879210 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:24.884518 (+ 5308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.885702 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.885707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.885711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.885885 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:24.886402 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:24.886407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:24.886410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:24.886746 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:24.886943 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:24.888592 (+ 1649us) server_negotiation.cc:299] Negotiation successful
0619 15:03:24.888845 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 15:03:24.891126 31127 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.103.125:44261: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53"
member_type: NON_VOTER
last_known_addr {
host: "127.24.103.125"
port: 44261
}
attrs {
promote: true
}
}
dest_uuid: "0e27d27671e74cb798fe7148e4eb0ce8"
cas_config_opid_index: -1
I20260619 15:03:24.892242 31127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } }
I20260619 15:03:24.896134 31685 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8fe5da68b9e34c32aa5758e08a0bdd53 (127.24.103.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:24.897558 31630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.124.keytab.
Entry for principal oryx/127.24.103.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.124.keytab.
I20260619 15:03:24.900391 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:39403
--webserver_interface=127.24.103.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.124.keytab
--principal=oryx/127.24.103.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:36187,127.24.103.125:44261,127.24.103.124:39403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:03:24.902307 31630 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 FOLLOWER]: Refusing update from remote peer 0e27d27671e74cb798fe7148e4eb0ce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 15:03:24.903487 31686 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:24.910934 31685 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.103.125:44261
I20260619 15:03:24.913722 31588 master.cc:537] Master@127.24.103.125:44261 shutting down...
I20260619 15:03:24.915346 31686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:24.915664 31686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:24.968690 31630 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8fe5da68b9e34c32aa5758e08a0bdd53 (127.24.103.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:24.973709 31695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:24.974149 31695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: This master's current role is: LEARNER
I20260619 15:03:24.974061 31662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e27d27671e74cb798fe7148e4eb0ce8. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:24.974341 31662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: This master's current role is: LEARNER
I20260619 15:03:24.974838 31692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:24.975186 31692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: This master's current role is: LEARNER
I20260619 15:03:24.978063 31588 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 LEARNER]: Raft consensus shutting down.
I20260619 15:03:24.978579 31588 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 LEARNER]: Raft consensus is shut down!
I20260619 15:03:24.978739 31588 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: stopping tablet replica
I20260619 15:03:24.996778 31588 master.cc:552] Master@127.24.103.125:44261 shutdown complete.
I20260619 15:03:24.996892 31588 master_runner.cc:305] Clearing existing system tablet
I20260619 15:03:24.997318 31588 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:24.997419 31588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:25.000464 31588 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 15:03:25.002285 31697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:25.002686 31588 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:25.002998 31588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 15:03:25.003085 31588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/wal
uuid: "8fe5da68b9e34c32aa5758e08a0bdd53"
format_stamp: "Formatted at 2026-06-19 15:03:24 on dist-test-slave-zpbw"
server_key: "990f7dd31dead5dc2e47fa4670733a83"
server_key_iv: "5daf4f5ac75cac07e018e7356a7f52aa"
server_key_version: "encryptionkey@0"
I20260619 15:03:25.007969 31588 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:25.014920 31588 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 15:03:25.015115 31588 log.cc:1196] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 15:03:25.015661 31588 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: Deleting consensus metadata
I20260619 15:03:25.015928 31588 master_runner.cc:315] Copying system tablet from 127.24.103.126:36187
I20260619 15:03:25.018561 31588 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: tablet copy: Beginning tablet copy session from remote peer at address 127.24.103.126:36187
I20260619 15:03:25.035161 31659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.019472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.24.103.126:36187)
0619 15:03:25.019667 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:25.019674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:25.020174 (+ 500us) server_negotiation.cc:407] Connection header received
0619 15:03:25.020362 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:25.020367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:25.020453 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:25.020582 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:25.021556 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.022679 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.024596 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.025252 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.025386 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.027833 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:25.027840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:25.027844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:25.027886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:25.032003 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.032885 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.032890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.032894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.033050 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.033514 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.033518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.033521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.033916 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:25.034072 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:25.034503 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 15:03:25.034731 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 15:03:25.035270 31703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.019361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:58738)
0619 15:03:25.020033 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:25.020061 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:25.020217 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:25.020609 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:25.020614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:25.021121 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:25.021370 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.021382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.022889 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.022895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.024420 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.024436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.025432 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.025439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.025655 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.026388 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:25.026421 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:25.027608 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:25.032168 (+ 4560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.032174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.032178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.032741 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.033192 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.033197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.033219 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.033387 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.034046 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:25.034051 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:25.034238 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 15:03:25.034393 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 15:03:25.034771 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":404,"threads_started":1}
I20260619 15:03:25.036835 31147 tablet_copy_service.cc:140] P 0e27d27671e74cb798fe7148e4eb0ce8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8fe5da68b9e34c32aa5758e08a0bdd53 ({username='oryx', principal='oryx/127.24.103.125@KRBTEST.COM'} at 127.0.0.1:58738)
I20260619 15:03:25.037154 31147 tablet_copy_service.cc:161] P 0e27d27671e74cb798fe7148e4eb0ce8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8fe5da68b9e34c32aa5758e08a0bdd53 at {username='oryx', principal='oryx/127.24.103.125@KRBTEST.COM'} at 127.0.0.1:58738: session id = 8fe5da68b9e34c32aa5758e08a0bdd53-00000000000000000000000000000000
I20260619 15:03:25.041944 31147 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 15:03:25.044420 31588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 15:03:25.051568 31588 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: tablet copy: Starting download of 0 data blocks...
I20260619 15:03:25.051937 31588 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: tablet copy: Starting download of 1 WAL segments...
I20260619 15:03:25.055068 31588 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 15:03:25.060123 31147 tablet_copy_service.cc:342] P 0e27d27671e74cb798fe7148e4eb0ce8: Request end of tablet copy session 8fe5da68b9e34c32aa5758e08a0bdd53-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.103.125@KRBTEST.COM'} at 127.0.0.1:58738
I20260619 15:03:25.060443 31147 tablet_copy_service.cc:433] P 0e27d27671e74cb798fe7148e4eb0ce8: ending tablet copy session 8fe5da68b9e34c32aa5758e08a0bdd53-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8fe5da68b9e34c32aa5758e08a0bdd53
I20260619 15:03:25.072559 31588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:25.075492 31704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:25.080210 31705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:25.091178 31588 server_base.cc:1034] running on GCE node
W20260619 15:03:25.091236 31707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:25.092020 31588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:25.092708 31588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:25.093930 31588 hybrid_clock.cc:648] HybridClock initialized: now 1781881405093859 us; error 46 us; skew 500 ppm
I20260619 15:03:25.094226 31588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:25 dist-test-slave-zpbw krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881405, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:25.096565 31588 init.cc:377] Logged in from keytab as oryx/127.24.103.125@KRBTEST.COM (short username oryx)
I20260619 15:03:25.097939 31588 webserver.cc:460] Webserver started at http://127.24.103.125:33447/ using document root <none> and password file <none>
I20260619 15:03:25.098305 31588 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:25.098414 31588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:25.101694 31588 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:03:25.103811 31714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:25.104316 31588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 15:03:25.104446 31588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/wal
uuid: "8fe5da68b9e34c32aa5758e08a0bdd53"
format_stamp: "Formatted at 2026-06-19 15:03:24 on dist-test-slave-zpbw"
server_key: "990f7dd31dead5dc2e47fa4670733a83"
server_key_iv: "5daf4f5ac75cac07e018e7356a7f52aa"
server_key_version: "encryptionkey@0"
I20260619 15:03:25.104610 31588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:25.125749 31588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:25.130548 31588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:25.174268 31588 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.125:44261
I20260619 15:03:25.174293 31765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.125:44261 every 8 connection(s)
I20260619 15:03:25.175670 31588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
I20260619 15:03:25.181723 31766 sys_catalog.cc:263] Verifying existing consensus state
I20260619 15:03:25.186754 31766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: Bootstrap starting.
I20260619 15:03:25.203208 31766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:03:25.203838 31766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: Bootstrap complete.
I20260619 15:03:25.205801 31766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } }
I20260619 15:03:25.206075 31766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fe5da68b9e34c32aa5758e08a0bdd53, State: Initialized, Role: LEARNER
I20260619 15:03:25.206367 31766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } }
I20260619 15:03:25.207266 31769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:25.207616 31766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:25.207686 31769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: This master's current role is: LEARNER
I20260619 15:03:25.219327 31780 catalog_manager.cc:1259] Loaded cluster ID: b9aec6e2a07b498ba36c28cb897b5765
I20260619 15:03:25.219563 31780 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: loading cluster ID for follower catalog manager: success
I20260619 15:03:25.223452 31780 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: acquiring CA information for follower catalog manager: success
I20260619 15:03:25.226101 31780 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 15:03:25.261080 31583 heartbeater.cc:503] Master 127.24.103.126:36187 was elected leader, sending a full tablet report...
W20260619 15:03:25.310861 31688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:25.311162 31688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:25.311323 31688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:25.319567 31688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:25.319694 31688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:25.319780 31688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:25.319885 31688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:25.319944 31688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:03:25.330294 31688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:36187,127.24.103.125:44261,127.24.103.124:39403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.124.keytab
--principal=oryx/127.24.103.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:39403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.103.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:25.331764 31688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:25.333150 31688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:25.345841 31788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:25.345928 31786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:25.345903 31785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:25.346858 31688 server_base.cc:1034] running on GCE node
I20260619 15:03:25.347694 31688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:25.349129 31688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:25.350327 31688 hybrid_clock.cc:648] HybridClock initialized: now 1781881405350285 us; error 41 us; skew 500 ppm
I20260619 15:03:25.350703 31688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:25 dist-test-slave-zpbw krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881405, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:25.360105 31688 init.cc:377] Logged in from keytab as oryx/127.24.103.124@KRBTEST.COM (short username oryx)
I20260619 15:03:25.361821 31688 webserver.cc:460] Webserver started at http://127.24.103.124:43891/ using document root <none> and password file <none>
I20260619 15:03:25.362571 31688 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:25.362720 31688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:25.363130 31688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:25.366490 31688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data/instance:
uuid: "45ddcfea2edc41979fcb0a55eb3662d7"
format_stamp: "Formatted at 2026-06-19 15:03:25 on dist-test-slave-zpbw"
server_key: "6f0b05fc1d42833c4a3f304b6c7deaff"
server_key_iv: "5a63f758abf160b2a5925ea6125edc7c"
server_key_version: "encryptionkey@0"
I20260619 15:03:25.367501 31688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/wal/instance:
uuid: "45ddcfea2edc41979fcb0a55eb3662d7"
format_stamp: "Formatted at 2026-06-19 15:03:25 on dist-test-slave-zpbw"
server_key: "6f0b05fc1d42833c4a3f304b6c7deaff"
server_key_iv: "5a63f758abf160b2a5925ea6125edc7c"
server_key_version: "encryptionkey@0"
I20260619 15:03:25.373029 31688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:03:25.376740 31795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:25.378274 31688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:03:25.378468 31688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/wal
uuid: "45ddcfea2edc41979fcb0a55eb3662d7"
format_stamp: "Formatted at 2026-06-19 15:03:25 on dist-test-slave-zpbw"
server_key: "6f0b05fc1d42833c4a3f304b6c7deaff"
server_key_iv: "5a63f758abf160b2a5925ea6125edc7c"
server_key_version: "encryptionkey@0"
I20260619 15:03:25.378681 31688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:25.408679 31688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:25.413441 31688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:25.413781 31688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:25.434393 31688 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.124:39403
I20260619 15:03:25.434417 31846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.124:39403 every 8 connection(s)
I20260619 15:03:25.435933 31688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
I20260619 15:03:25.439265 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 31688
I20260619 15:03:25.440415 31847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:25.445006 31686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:25.445302 31686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:25.452435 31847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:03:25 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.124@KRBTEST.COM
I20260619 15:03:25.473207 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.445981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.24.103.125:44261)
0619 15:03:25.465707 (+ 19726us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:25.465713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:25.465734 (+ 21us) server_negotiation.cc:407] Connection header received
0619 15:03:25.465807 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:25.465813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:25.465915 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:25.466042 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:25.466731 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.468425 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.470341 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.471996 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.472195 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.472399 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:25.472678 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 15:03:25.472860 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19577,"thread_start_us":19881,"threads_started":1}
I20260619 15:03:25.473251 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.445780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:44261 (local address 127.0.0.1:50534)
0619 15:03:25.464336 (+ 18556us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:25.464368 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:25.464544 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:25.466301 (+ 1757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:25.466307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:25.466340 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:25.466552 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.466564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.468592 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.468599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.470153 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.470171 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.472199 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.472207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.472441 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.472519 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 15:03:25.472652 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 15:03:25.472837 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18315,"mutex_wait_us":23,"thread_start_us":18134,"threads_started":1}
I20260619 15:03:25.474275 31769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e27d27671e74cb798fe7148e4eb0ce8. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:25.474593 31769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: This master's current role is: LEARNER
I20260619 15:03:25.475389 31686 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8: attempting to promote NON_VOTER 8fe5da68b9e34c32aa5758e08a0bdd53 to VOTER
I20260619 15:03:25.476457 31686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } }
Jun 19 15:03:25 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881405, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.124@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:25.478375 31741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 LEARNER]: Refusing update from remote peer 0e27d27671e74cb798fe7148e4eb0ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 15:03:25.479022 31852 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 15:03:25.483933 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.441436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.24.103.124:39403)
0619 15:03:25.442200 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:25.442212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:25.442253 (+ 41us) server_negotiation.cc:407] Connection header received
0619 15:03:25.443887 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:25.443909 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:25.444405 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:25.444860 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:25.455008 (+ 10148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.457894 (+ 2886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.465010 (+ 7116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.466922 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.467164 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.473141 (+ 5977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:25.473154 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:25.473171 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:25.473229 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:25.477741 (+ 4512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.478687 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.478699 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.478710 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.478969 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.479471 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.479475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.479478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.480275 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:25.480560 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:25.481137 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 15:03:25.481951 (+ 814us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":333,"threads_started":1}
I20260619 15:03:25.484905 31852 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fe5da68b9e34c32aa5758e08a0bdd53 (127.24.103.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } }
I20260619 15:03:25.486079 31659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.458019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.24.103.126:36187)
0619 15:03:25.458260 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:25.458268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:25.466052 (+ 7784us) server_negotiation.cc:407] Connection header received
0619 15:03:25.466369 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:25.466375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:25.466468 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:25.466580 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:25.468432 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.470009 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.472023 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.472664 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.472781 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.478687 (+ 5906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:25.478694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:25.478702 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:25.478742 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:25.482389 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.483435 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.483440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.483444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.483647 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.484097 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.484102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.484105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.484453 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:25.484648 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:25.485191 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 15:03:25.485414 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":81}
I20260619 15:03:25.485463 31741 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fe5da68b9e34c32aa5758e08a0bdd53 (127.24.103.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } }
I20260619 15:03:25.488628 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.457249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:58744)
0619 15:03:25.465858 (+ 8609us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:25.465918 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:25.466208 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:25.466956 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:25.466965 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:25.467923 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:25.468246 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.468258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.470197 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.470203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.471866 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.471882 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.473075 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.473083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.473642 (+ 559us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.475087 (+ 1445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:25.475111 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:25.478469 (+ 3358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:25.482546 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.482557 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.482564 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.483294 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.483774 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.483778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.483781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.483987 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.484607 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:25.484615 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:25.484822 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 15:03:25.487761 (+ 2939us) client_negotiation.cc:240] Negotiation successful
0619 15:03:25.488138 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8330,"thread_start_us":8171,"threads_started":1}
I20260619 15:03:25.489001 31686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } }
I20260619 15:03:25.489307 31686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:25.489662 31686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } }
I20260619 15:03:25.489894 31686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:25.489753 31769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } }
I20260619 15:03:25.489977 31769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:03:25.490365 31847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:03:25 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881405, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.124@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
I20260619 15:03:25.502835 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.490984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:44261 (local address 127.0.0.1:50550)
0619 15:03:25.491168 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:25.491187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:25.491293 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:25.491624 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:25.491627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:25.491940 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:25.492096 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.492104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.493114 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.493117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.494055 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.494066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.494699 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.494703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.494921 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.495472 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:25.495491 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:25.497441 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:25.500663 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.500667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.500669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.501075 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.501422 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.501425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.501427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.501556 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.501963 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:25.501966 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:25.502106 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 15:03:25.502221 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 15:03:25.502466 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 15:03:25.502847 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.491205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.24.103.125:44261)
0619 15:03:25.491379 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:25.491383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:25.491398 (+ 15us) server_negotiation.cc:407] Connection header received
0619 15:03:25.491442 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:25.491446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:25.491502 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:25.491581 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:25.492218 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.492984 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.494182 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.494579 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.494666 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.497615 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:25.497619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:25.497623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:25.497652 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:25.500455 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.501189 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.501193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.501195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.501332 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.501640 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.501643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.501645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.501875 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:25.501978 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:25.502304 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 15:03:25.502540 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 15:03:25.503857 31847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:03:25 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881405, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.124@KRBTEST.COM for oryx/127.24.103.124@KRBTEST.COM
I20260619 15:03:25.516017 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.504479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39403 (local address 127.0.0.1:36908)
0619 15:03:25.504667 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:25.504685 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:25.504819 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:25.505101 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:25.505105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:25.505473 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:25.505603 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.505610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.506698 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.506706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.507685 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.507696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.508414 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.508419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.508575 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.509129 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:25.509146 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:25.511119 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:25.513933 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.513937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.513940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.514287 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.514586 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.514589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.514591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.514728 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.515268 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:25.515271 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:25.515414 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 15:03:25.515516 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 15:03:25.515719 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 15:03:25.516103 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.504680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.24.103.124:39403)
0619 15:03:25.504819 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:25.504825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:25.504840 (+ 15us) server_negotiation.cc:407] Connection header received
0619 15:03:25.504923 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:25.504928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:25.504986 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:25.505078 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:25.505726 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.506577 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.507813 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.508215 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.508370 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.511302 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:25.511306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:25.511310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:25.511337 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:25.513815 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.514383 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.514387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.514389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.514496 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.514895 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.514898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.514900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.515171 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:25.515274 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:25.515614 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 15:03:25.515850 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":52}
I20260619 15:03:25.520785 31847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: Bootstrap starting.
I20260619 15:03:25.524050 31847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:25.525555 31847 log.cc:824] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:25.528357 31847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: No bootstrap required, opened a new log
I20260619 15:03:25.535595 31847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } }
I20260619 15:03:25.535931 31847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:25.536072 31847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ddcfea2edc41979fcb0a55eb3662d7, State: Initialized, Role: FOLLOWER
I20260619 15:03:25.536736 31847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } }
I20260619 15:03:25.537624 31859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } } }
I20260619 15:03:25.537912 31859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:03:25.538862 31847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 15:03:25.549294 31870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:03:25.549450 31870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:03:25.552982 31688 master_runner.cc:428] Detected that this master 45ddcfea2edc41979fcb0a55eb3662d7 is joining an existing cluster
I20260619 15:03:25.553098 31688 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.103.124:39403
I20260619 15:03:25.575263 31659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.557251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.24.103.126:36187)
0619 15:03:25.557481 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:25.557489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:25.558574 (+ 1085us) server_negotiation.cc:407] Connection header received
0619 15:03:25.558770 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:25.558798 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:25.558895 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:25.559010 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:25.560566 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.561803 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.563960 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.564602 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.564748 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.567707 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:25.567714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:25.567719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:25.567761 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:25.571738 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.572607 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.572614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.572618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.572783 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.573355 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.573360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.573363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.573711 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:25.573838 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:25.574551 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 15:03:25.574846 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 15:03:25.576818 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.557302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:58748)
0619 15:03:25.558451 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:25.558473 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:25.558627 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:25.559209 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:25.559216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:25.559987 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:25.560368 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.560382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.562031 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.562037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.563780 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.563795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.565082 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.565089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.565348 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.566126 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:25.566157 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:25.567483 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:25.571924 (+ 4441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.571930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.571934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.572464 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.572969 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.572974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.572978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.573195 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.573996 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:25.574002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:25.574213 (+ 211us) client_negotiation.cc:769] Sending connection context
0619 15:03:25.575832 (+ 1619us) client_negotiation.cc:240] Negotiation successful
0619 15:03:25.576274 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"thread_start_us":839,"threads_started":1}
I20260619 15:03:25.577410 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.558878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.24.103.125:44261)
0619 15:03:25.559059 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:25.559064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:25.559871 (+ 807us) server_negotiation.cc:407] Connection header received
0619 15:03:25.560298 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:25.560306 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:25.560407 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:25.560491 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:25.562418 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.563818 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.566530 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.567202 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.567326 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.570211 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:25.570217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:25.570222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:25.570266 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:25.573698 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.574498 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.574507 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.574513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.574679 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.575226 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.575231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.575235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.575607 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:25.575724 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:25.576345 (+ 621us) server_negotiation.cc:299] Negotiation successful
0619 15:03:25.576592 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 15:03:25.578593 31877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.559073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:44261 (local address 127.0.0.1:50564)
0619 15:03:25.559778 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:25.559794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:25.560108 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:25.560751 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:25.560758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:25.561691 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:25.562191 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.562207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.564041 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.564048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.566341 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.566359 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.567661 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.567670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.568283 (+ 613us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.568861 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:25.568882 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:25.570006 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:25.573850 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.573856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.573862 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.574342 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.574871 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.574878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.574884 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.575066 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.575822 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:25.575827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:25.576039 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 15:03:25.577684 (+ 1645us) client_negotiation.cc:240] Negotiation successful
0619 15:03:25.578049 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":3,"thread_start_us":369,"threads_started":1}
I20260619 15:03:25.584818 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.559888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.24.103.124:39403)
0619 15:03:25.560028 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:25.560033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:25.567043 (+ 7010us) server_negotiation.cc:407] Connection header received
0619 15:03:25.567401 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:25.567406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:25.567472 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:25.567539 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:25.569466 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.570565 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.572703 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.573462 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.573563 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.576446 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:25.576452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:25.576456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:25.576489 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:25.580705 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.581564 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.581570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.581574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.581770 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.582160 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.582166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.582169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.582617 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:25.582800 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:25.583330 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 15:03:25.584380 (+ 1050us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":781,"server-negotiator.queue_time_us":30}
I20260619 15:03:25.584896 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.560027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39403 (local address 127.0.0.1:36922)
0619 15:03:25.566951 (+ 6924us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:25.566970 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:25.567285 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:25.567724 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:25.567729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:25.568893 (+ 1164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:25.569269 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.569282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.570825 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.570834 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.572551 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.572565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.573914 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.573925 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.574183 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.574954 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:25.574977 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:25.576277 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:25.580863 (+ 4586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.580867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.580871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.581410 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.581888 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.581893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.581896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.582046 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.582887 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:25.582892 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:25.583071 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 15:03:25.583235 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 15:03:25.583553 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6781,"spinlock_wait_cycles":94592,"thread_start_us":7040,"threads_started":1}
I20260619 15:03:25.591766 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 31188
Jun 19 15:03:25 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881402, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.126@KRBTEST.COM for oryx/127.24.103.124@KRBTEST.COM
I20260619 15:03:25.601372 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.583802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39403 (local address 127.0.0.1:36930)
0619 15:03:25.584316 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:25.584332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:25.584473 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:25.585306 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:25.585313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:25.585818 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:25.586017 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.586029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.587437 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.587443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.588330 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.588342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.589217 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.589225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.589529 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.590292 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:25.590314 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:25.594246 (+ 3932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:25.598385 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.598391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.598395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.599044 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.599473 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.599478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.599482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.599655 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.600336 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:25.600344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:25.600547 (+ 203us) client_negotiation.cc:769] Sending connection context
0619 15:03:25.600712 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 15:03:25.601006 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379}
I20260619 15:03:25.601426 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.584091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.24.103.124:39403)
0619 15:03:25.584909 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:25.584916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:25.584941 (+ 25us) server_negotiation.cc:407] Connection header received
0619 15:03:25.585049 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:25.585057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:25.585143 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:25.585247 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:25.586225 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.587306 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.588474 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.589007 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.589130 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.594492 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:25.594500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:25.594505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:25.594556 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:25.597905 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.599180 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.599187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.599191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.599339 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.599798 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.599803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.599808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.600202 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:25.600351 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:25.600797 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 15:03:25.601006 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":420,"threads_started":1}
I20260619 15:03:25.603328 31127 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.103.124:39403: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7"
member_type: NON_VOTER
last_known_addr {
host: "127.24.103.124"
port: 39403
}
attrs {
promote: true
}
}
dest_uuid: "0e27d27671e74cb798fe7148e4eb0ce8"
cas_config_opid_index: 6
I20260619 15:03:25.604166 31127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true } }
Jun 19 15:03:25 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.65@KRBTEST.COM
I20260619 15:03:25.606393 31741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 FOLLOWER]: Refusing update from remote peer 0e27d27671e74cb798fe7148e4eb0ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 15:03:25.607261 31822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:25.607185 31686 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 15:03:25.609851 31851 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 45ddcfea2edc41979fcb0a55eb3662d7 (127.24.103.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true } } }
I20260619 15:03:25.610317 31741 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 45ddcfea2edc41979fcb0a55eb3662d7 (127.24.103.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true } } }
I20260619 15:03:25.610996 31822 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 FOLLOWER]: Refusing update from remote peer 0e27d27671e74cb798fe7148e4eb0ce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 15:03:25.611868 31852 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:25.614183 31851 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.103.124:39403
I20260619 15:03:25.614478 31851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true } } }
I20260619 15:03:25.614596 31769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true } } }
I20260619 15:03:25.614847 31851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:25.614886 31769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:03:25.615067 31852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true } } }
I20260619 15:03:25.615339 31852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:25.616816 31688 master.cc:537] Master@127.24.103.124:39403 shutting down...
I20260619 15:03:25.623919 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.592819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.24.103.65:38491)
0619 15:03:25.593354 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:25.593361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:25.593387 (+ 26us) server_negotiation.cc:407] Connection header received
0619 15:03:25.593444 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:25.593451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:25.593673 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:25.593841 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:25.595590 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.597214 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.598813 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.599591 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.599721 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.606720 (+ 6999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:25.606743 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:25.606762 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:25.607901 (+ 1139us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:25.618197 (+ 10296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.619648 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.619658 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.619662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.619830 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.620416 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.620424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.620430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.621018 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:25.621392 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:25.622972 (+ 1580us) server_negotiation.cc:299] Negotiation successful
0619 15:03:25.623325 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":566,"threads_started":1}
I20260619 15:03:25.625921 31311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:25.635633 31821 raft_consensus.cc:858] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } }
W20260619 15:03:25.635862 31821 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } }
I20260619 15:03:25.663853 31821 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8fe5da68b9e34c32aa5758e08a0bdd53 (127.24.103.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:25.668881 31859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e27d27671e74cb798fe7148e4eb0ce8. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:25.669148 31859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 15:03:25.669564 31887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:25.669951 31887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 15:03:25.670374 31851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true } } }
I20260619 15:03:25.670637 31851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:25.671013 31884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: true } } }
I20260619 15:03:25.671761 31884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: This master's current role is: NON_PARTICIPANT
W20260619 15:03:25.671738 31114 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 15:03:25.675205 31688 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 15:03:25.675851 31688 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 15:03:25.676061 31688 tablet_replica.cc:330] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: stopping tablet replica
W20260619 15:03:25.682018 31114 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 -> Peer 45ddcfea2edc41979fcb0a55eb3662d7 (127.24.103.124:39403): Couldn't send request to peer 45ddcfea2edc41979fcb0a55eb3662d7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 15:03:25.695391 31688 master.cc:552] Master@127.24.103.124:39403 shutdown complete.
I20260619 15:03:25.695508 31688 master_runner.cc:305] Clearing existing system tablet
I20260619 15:03:25.695855 31688 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:25.695969 31688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:25.698251 31688 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 15:03:25.699941 31890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:25.700346 31688 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:25.700598 31688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 15:03:25.700692 31688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/wal
uuid: "45ddcfea2edc41979fcb0a55eb3662d7"
format_stamp: "Formatted at 2026-06-19 15:03:25 on dist-test-slave-zpbw"
server_key: "6f0b05fc1d42833c4a3f304b6c7deaff"
server_key_iv: "5a63f758abf160b2a5925ea6125edc7c"
server_key_version: "encryptionkey@0"
I20260619 15:03:25.704187 31688 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:25.709441 31688 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 15:03:25.709635 31688 log.cc:1196] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 15:03:25.710144 31688 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: Deleting consensus metadata
I20260619 15:03:25.710381 31688 master_runner.cc:315] Copying system tablet from 127.24.103.126:36187
I20260619 15:03:25.718636 31688 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: tablet copy: Beginning tablet copy session from remote peer at address 127.24.103.126:36187
I20260619 15:03:25.736027 31659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.719600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58752 (local address 127.24.103.126:36187)
0619 15:03:25.719891 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:25.719900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:25.723867 (+ 3967us) server_negotiation.cc:407] Connection header received
0619 15:03:25.724093 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:25.724099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:25.724191 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:25.724299 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:25.725132 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.725878 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.727200 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.727636 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.727768 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.729672 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:25.729678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:25.729681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:25.729714 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:25.732875 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.733755 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.733762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.733765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.733914 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:25.734350 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:25.734355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:25.734357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:25.734720 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:25.734888 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:25.735508 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 15:03:25.735777 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260619 15:03:25.736948 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:25.719497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:58752)
0619 15:03:25.723706 (+ 4209us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:25.723727 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:25.723940 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:25.724326 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:25.724330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:25.724782 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:25.724976 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.724985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.726038 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.726050 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.727048 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:25.727059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:25.727792 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:25.727797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:25.727984 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:25.728668 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:25.728685 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:25.729445 (+ 760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:25.733066 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.733072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.733076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.733609 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.734065 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:25.734070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:25.734074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:25.734238 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:25.735003 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:25.735009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:25.735237 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 15:03:25.736248 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0619 15:03:25.736632 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4086,"thread_start_us":3506,"threads_started":1}
I20260619 15:03:25.737710 31147 tablet_copy_service.cc:140] P 0e27d27671e74cb798fe7148e4eb0ce8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 45ddcfea2edc41979fcb0a55eb3662d7 ({username='oryx', principal='oryx/127.24.103.124@KRBTEST.COM'} at 127.0.0.1:58752)
I20260619 15:03:25.737879 31147 tablet_copy_service.cc:161] P 0e27d27671e74cb798fe7148e4eb0ce8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 45ddcfea2edc41979fcb0a55eb3662d7 at {username='oryx', principal='oryx/127.24.103.124@KRBTEST.COM'} at 127.0.0.1:58752: session id = 45ddcfea2edc41979fcb0a55eb3662d7-00000000000000000000000000000000
I20260619 15:03:25.741139 31147 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 15:03:25.743248 31688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 15:03:25.748027 31688 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: tablet copy: Starting download of 0 data blocks...
I20260619 15:03:25.748345 31688 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: tablet copy: Starting download of 1 WAL segments...
I20260619 15:03:25.750665 31688 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 15:03:25.754303 31147 tablet_copy_service.cc:342] P 0e27d27671e74cb798fe7148e4eb0ce8: Request end of tablet copy session 45ddcfea2edc41979fcb0a55eb3662d7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.103.124@KRBTEST.COM'} at 127.0.0.1:58752
I20260619 15:03:25.754498 31147 tablet_copy_service.cc:433] P 0e27d27671e74cb798fe7148e4eb0ce8: ending tablet copy session 45ddcfea2edc41979fcb0a55eb3662d7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 45ddcfea2edc41979fcb0a55eb3662d7
I20260619 15:03:25.766431 31688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:25.771613 31897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:25.778288 31898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:25.782197 31900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:25.782812 31688 server_base.cc:1034] running on GCE node
I20260619 15:03:25.783128 31688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:25.783751 31688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:25.785004 31688 hybrid_clock.cc:648] HybridClock initialized: now 1781881405784973 us; error 27 us; skew 500 ppm
I20260619 15:03:25.785241 31688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:25 dist-test-slave-zpbw krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881405, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:25.787727 31688 init.cc:377] Logged in from keytab as oryx/127.24.103.124@KRBTEST.COM (short username oryx)
I20260619 15:03:25.789610 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 31188
I20260619 15:03:25.795817 31688 webserver.cc:460] Webserver started at http://127.24.103.124:35697/ using document root <none> and password file <none>
I20260619 15:03:25.796209 31688 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:25.796319 31688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:25.799741 31688 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 15:03:25.801890 31907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:25.802372 31688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 15:03:25.802505 31688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/wal
uuid: "45ddcfea2edc41979fcb0a55eb3662d7"
format_stamp: "Formatted at 2026-06-19 15:03:25 on dist-test-slave-zpbw"
server_key: "6f0b05fc1d42833c4a3f304b6c7deaff"
server_key_iv: "5a63f758abf160b2a5925ea6125edc7c"
server_key_version: "encryptionkey@0"
I20260619 15:03:25.802709 31688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:25.815227 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:38491
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:36187
--webserver_port=37031
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.103.126:36187,127.24.103.125:44261
--tserver_master_addrs=127.24.103.126:36187,127.24.103.125:44261,127.24.103.124:39403 with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:03:25.815409 31688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:25.819893 31688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:25.840036 31688 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.124:39403
I20260619 15:03:25.840077 31959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.124:39403 every 8 connection(s)
I20260619 15:03:25.841288 31688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
I20260619 15:03:25.846575 31960 sys_catalog.cc:263] Verifying existing consensus state
I20260619 15:03:25.850648 31960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: Bootstrap starting.
I20260619 15:03:25.865444 31960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:03:25.865969 31960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: Bootstrap complete.
I20260619 15:03:25.867029 31960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true } }
I20260619 15:03:25.867336 31960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 45ddcfea2edc41979fcb0a55eb3662d7, State: Initialized, Role: LEARNER
I20260619 15:03:25.867569 31960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true } }
I20260619 15:03:25.868639 31960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:25.868387 31963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true } } }
I20260619 15:03:25.868904 31963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: This master's current role is: LEARNER
I20260619 15:03:25.885053 31974 catalog_manager.cc:1259] Loaded cluster ID: b9aec6e2a07b498ba36c28cb897b5765
I20260619 15:03:25.885185 31974 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: loading cluster ID for follower catalog manager: success
I20260619 15:03:25.887796 31974 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: acquiring CA information for follower catalog manager: success
I20260619 15:03:25.889695 31974 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 15:03:26.051007 31686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true } } }
I20260619 15:03:26.051367 31686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:26.058001 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:26.051435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39403 (local address 127.0.0.1:36932)
0619 15:03:26.051835 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:26.051858 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:26.052004 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:26.053007 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:26.053013 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:26.053043 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:26.053250 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:26.053260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.054682 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.054688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:26.055911 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:26.055923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.057122 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.057129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:26.057346 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:26.057379 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 15:03:26.057483 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 15:03:26.057656 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260619 15:03:26.058058 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:26.051991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.24.103.124:39403)
0619 15:03:26.052657 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:26.052661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:26.052681 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:03:26.052745 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:26.052749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:26.052817 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:26.052971 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:26.053420 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.054522 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.056088 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.056941 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.057165 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:26.057380 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:26.057520 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 15:03:26.057658 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":390,"threads_started":1}
I20260619 15:03:26.058986 31935 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260619 15:03:26.059706 31963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e27d27671e74cb798fe7148e4eb0ce8. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: true } } }
I20260619 15:03:26.060039 31963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: This master's current role is: LEARNER
I20260619 15:03:26.060230 31851 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8: attempting to promote NON_VOTER 45ddcfea2edc41979fcb0a55eb3662d7 to VOTER
I20260619 15:03:26.060887 31851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } }
I20260619 15:03:26.062914 31935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 LEARNER]: Refusing update from remote peer 0e27d27671e74cb798fe7148e4eb0ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 15:03:26.062959 31741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 FOLLOWER]: Refusing update from remote peer 0e27d27671e74cb798fe7148e4eb0ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 15:03:26.063547 31686 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 15:03:26.063798 31852 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 15:03:26.067142 31686 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 45ddcfea2edc41979fcb0a55eb3662d7 (127.24.103.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } } }
I20260619 15:03:26.069507 31741 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 45ddcfea2edc41979fcb0a55eb3662d7 (127.24.103.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } } }
I20260619 15:03:26.080116 31769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } } }
I20260619 15:03:26.080407 31769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:03:26.080830 31851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } } }
I20260619 15:03:26.081207 31851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:26.081629 31851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } } }
I20260619 15:03:26.081923 31851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:26.083873 31851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } } }
I20260619 15:03:26.084189 31851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:26.084681 31935 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 45ddcfea2edc41979fcb0a55eb3662d7 (127.24.103.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } } }
I20260619 15:03:26.087744 31963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } } }
I20260619 15:03:26.087994 31963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 15:03:26.252351 31914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:26.252664 31914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:26.252826 31914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:26.260411 31914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:26.260555 31914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:26.260746 31914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:03:26.271519 31914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:38491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=37031
--tserver_master_addrs=127.24.103.126:36187,127.24.103.125:44261,127.24.103.124:39403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:26.273010 31914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:26.274551 31914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:26.289054 31981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:26.289059 31982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:26.289075 31984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:26.290437 31914 server_base.cc:1034] running on GCE node
I20260619 15:03:26.291209 31914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:26.292145 31914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:26.293358 31914 hybrid_clock.cc:648] HybridClock initialized: now 1781881406293305 us; error 48 us; skew 500 ppm
I20260619 15:03:26.293696 31914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:26 dist-test-slave-zpbw krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881406, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:26.296953 31914 init.cc:377] Logged in from keytab as oryx/127.24.103.65@KRBTEST.COM (short username oryx)
I20260619 15:03:26.298439 31914 webserver.cc:460] Webserver started at http://127.24.103.65:37031/ using document root <none> and password file <none>
I20260619 15:03:26.299232 31914 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:26.299409 31914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:26.305330 31914 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 15:03:26.308436 31991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:26.309857 31914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:03:26.310058 31914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "76c90e3825b54f8f91d82c5ba5e9c102"
format_stamp: "Formatted at 2026-06-19 15:03:23 on dist-test-slave-zpbw"
server_key: "85c604d27c0b63ee49d1e5497b68355f"
server_key_iv: "fbdf1395631d94967b19a3156ccd6413"
server_key_version: "encryptionkey@0"
I20260619 15:03:26.311174 31914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:26.334199 31914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:26.338864 31914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:26.339192 31914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:26.340230 31914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:26.341732 31914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:26.341858 31914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:26.341987 31914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:26.342077 31914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:26.380604 31914 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:38491
I20260619 15:03:26.380657 32103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:38491 every 8 connection(s)
I20260619 15:03:26.382565 31914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:03:26.386616 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 31914
I20260619 15:03:26.386998 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 31322
Jun 19 15:03:26 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.66@KRBTEST.COM
Jun 19 15:03:26 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881406, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.124@KRBTEST.COM
Jun 19 15:03:26 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881406, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:26.426859 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:26.388053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.24.103.66:34037)
0619 15:03:26.401224 (+ 13171us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:26.401233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:26.401263 (+ 30us) server_negotiation.cc:407] Connection header received
0619 15:03:26.401333 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:26.401343 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:26.401689 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:26.401917 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:26.403938 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.405616 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.408078 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.409040 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.416274 (+ 7234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:26.416624 (+ 350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:26.416638 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:26.416656 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:26.416697 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:26.422455 (+ 5758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:26.423535 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:26.423544 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:26.423548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:26.423705 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:26.424189 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:26.424196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:26.424201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:26.424662 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:26.424946 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:26.426074 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0619 15:03:26.426385 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12984,"thread_start_us":12817,"threads_started":1}
I20260619 15:03:26.427277 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:26.399062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:46731 (local address 127.24.103.124:39403)
0619 15:03:26.399279 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:26.399288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:26.403600 (+ 4312us) server_negotiation.cc:407] Connection header received
0619 15:03:26.403786 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:26.403794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:26.403887 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:26.404009 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:26.406931 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.408694 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.411640 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.412410 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.412561 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:26.419423 (+ 6862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:26.419431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:26.419437 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:26.419490 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:26.423055 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:26.424367 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:26.424373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:26.424376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:26.424530 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:26.425092 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:26.425097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:26.425101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:26.425464 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:26.425590 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:26.426668 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0619 15:03:26.426937 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 15:03:26.427861 32108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:26.385471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:48471 (local address 127.24.103.126:36187)
0619 15:03:26.404860 (+ 19389us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:26.404870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:26.404895 (+ 25us) server_negotiation.cc:407] Connection header received
0619 15:03:26.405102 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:26.405110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:26.405211 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:26.405357 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:26.407194 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.409138 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.411674 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.412616 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.412764 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:26.419443 (+ 6679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:26.419453 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:26.419461 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:26.419517 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:26.423555 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:26.424465 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:26.424473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:26.424477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:26.424663 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:26.425105 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:26.425110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:26.425114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:26.425443 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:26.425577 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:26.427232 (+ 1655us) server_negotiation.cc:299] Negotiation successful
0619 15:03:26.427500 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19221,"thread_start_us":19463,"threads_started":1}
I20260619 15:03:26.428845 31445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:26.429142 32109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:26.386366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.65:48471)
0619 15:03:26.398115 (+ 11749us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:26.398186 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:26.400955 (+ 2769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:26.405681 (+ 4726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:26.405689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:26.406431 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:26.406932 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:26.406948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.409373 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.409381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:26.411415 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:26.411435 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.413098 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.413107 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:26.413494 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:26.414500 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:26.414529 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:26.419116 (+ 4587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:26.423720 (+ 4604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:26.423727 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:26.423732 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:26.424321 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:26.424797 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:26.424804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:26.424807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:26.424977 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:26.425662 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:26.425671 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:26.426249 (+ 578us) client_negotiation.cc:769] Sending connection context
0619 15:03:26.426539 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 15:03:26.427223 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11396,"mutex_wait_us":229,"spinlock_wait_cycles":25344,"thread_start_us":11116,"threads_started":1}
I20260619 15:03:26.430399 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:26.399024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39403 (local address 127.24.103.65:46731)
0619 15:03:26.403446 (+ 4422us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:26.403472 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:26.403638 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:26.404693 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:26.404705 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:26.405660 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:26.406673 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:26.406698 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.408912 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.408918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:26.411406 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:26.411437 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.412968 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.412975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:26.413488 (+ 513us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:26.414348 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:26.414389 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:26.419112 (+ 4723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:26.423236 (+ 4124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:26.423246 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:26.423265 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:26.424220 (+ 955us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:26.424682 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:26.424686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:26.424690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:26.424950 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:26.425669 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:26.425673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:26.426249 (+ 576us) client_negotiation.cc:769] Sending connection context
0619 15:03:26.426561 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 15:03:26.428781 (+ 2220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4239,"mutex_wait_us":1723,"spinlock_wait_cycles":9088,"thread_start_us":4688,"threads_started":1}
I20260619 15:03:26.431339 32105 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36187
I20260619 15:03:26.431661 32105 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:26.432816 32105 heartbeater.cc:511] Master 127.24.103.126:36187 requested a full tablet report, sending...
I20260619 15:03:26.433966 32104 heartbeater.cc:347] Connected to a master server at 127.24.103.124:39403
I20260619 15:03:26.434165 32104 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:26.434682 32104 heartbeater.cc:511] Master 127.24.103.124:39403 requested a full tablet report, sending...
I20260619 15:03:26.435200 31127 ts_manager.cc:194] Re-registered known tserver with Master: 76c90e3825b54f8f91d82c5ba5e9c102 (127.24.103.65:38491)
Jun 19 15:03:26 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881406, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
I20260619 15:03:26.436615 31925 ts_manager.cc:194] Registered new tserver with Master: 76c90e3825b54f8f91d82c5ba5e9c102 (127.24.103.65:38491)
I20260619 15:03:26.436781 31127 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:48471
I20260619 15:03:26.444449 32113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:26.404494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:48819 (local address 127.24.103.125:44261)
0619 15:03:26.422357 (+ 17863us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:26.422363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:26.422384 (+ 21us) server_negotiation.cc:407] Connection header received
0619 15:03:26.422450 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:26.422457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:26.422542 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:26.422654 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:26.424377 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.425634 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.430264 (+ 4630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.431025 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.431173 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:26.438525 (+ 7352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:26.438536 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:26.438540 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:26.438578 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:26.441670 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:26.442514 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:26.442518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:26.442520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:26.442643 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:26.443080 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:26.443083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:26.443085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:26.443361 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:26.443506 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:26.444018 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 15:03:26.444181 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17670,"thread_start_us":18970,"threads_started":1}
I20260619 15:03:26.444613 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:26.410482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:44261 (local address 127.24.103.65:48819)
0619 15:03:26.419969 (+ 9487us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:26.419989 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:26.420127 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:26.422892 (+ 2765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:26.422900 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:26.423649 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:26.424179 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:26.424196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.425788 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.425792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:26.430046 (+ 4254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:26.430066 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:26.431486 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:26.431492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:26.431748 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:26.432888 (+ 1140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:26.432985 (+ 97us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:26.437828 (+ 4843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:26.441834 (+ 4006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:26.441839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:26.441844 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:26.442354 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:26.442758 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:26.442764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:26.442768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:26.442967 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:26.443484 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:26.443489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:26.443642 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 15:03:26.443783 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 15:03:26.444190 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9349,"mutex_wait_us":896,"thread_start_us":9721,"threads_started":1}
I20260619 15:03:26.445521 32106 heartbeater.cc:347] Connected to a master server at 127.24.103.125:44261
I20260619 15:03:26.445680 32106 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:26.446081 32106 heartbeater.cc:511] Master 127.24.103.125:44261 requested a full tablet report, sending...
I20260619 15:03:26.447739 31731 ts_manager.cc:194] Registered new tserver with Master: 76c90e3825b54f8f91d82c5ba5e9c102 (127.24.103.65:38491)
I20260619 15:03:26.523963 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 31322
I20260619 15:03:26.540073 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:34037
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:36187
--webserver_port=41097
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.103.126:36187,127.24.103.125:44261
--tserver_master_addrs=127.24.103.126:36187,127.24.103.125:44261,127.24.103.124:39403 with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:26.888777 32116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:26.889199 32116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:26.889355 32116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:26.897265 32116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:26.897385 32116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:26.897576 32116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:03:26.908064 32116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:34037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=41097
--tserver_master_addrs=127.24.103.126:36187,127.24.103.125:44261,127.24.103.124:39403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:26.909614 32116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:26.911028 32116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:26.924630 32122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:26.924516 32121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:26.924559 32124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:26.932258 32116 server_base.cc:1034] running on GCE node
I20260619 15:03:26.932969 32116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:26.934000 32116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:26.935230 32116 hybrid_clock.cc:648] HybridClock initialized: now 1781881406935182 us; error 45 us; skew 500 ppm
I20260619 15:03:26.935557 32116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:26 dist-test-slave-zpbw krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881406, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:26.938846 32116 init.cc:377] Logged in from keytab as oryx/127.24.103.66@KRBTEST.COM (short username oryx)
I20260619 15:03:26.940380 32116 webserver.cc:460] Webserver started at http://127.24.103.66:41097/ using document root <none> and password file <none>
I20260619 15:03:26.941134 32116 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:26.941250 32116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:26.946965 32116 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 15:03:26.949586 32131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:26.951036 32116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:03:26.951200 32116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "638989ea9b4c4df599319b3f97e1b5d9"
format_stamp: "Formatted at 2026-06-19 15:03:23 on dist-test-slave-zpbw"
server_key: "cf78cb434b672d3cfbf3be5e522eaeb4"
server_key_iv: "30452809cdb5446ee52e5984f6934e63"
server_key_version: "encryptionkey@0"
I20260619 15:03:26.952200 32116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:26.983115 32116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:26.987756 32116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:26.988065 32116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:26.989063 32116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:26.990583 32116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:26.990682 32116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:26.990841 32116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:26.990926 32116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:27.024613 32116 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:34037
I20260619 15:03:27.024701 32243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:34037 every 8 connection(s)
I20260619 15:03:27.026126 32116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:03:27.032647 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 32116
I20260619 15:03:27.032976 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 31455
Jun 19 15:03:27 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881406, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.124@KRBTEST.COM
Jun 19 15:03:27 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881406, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
Jun 19 15:03:27 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881406, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
Jun 19 15:03:27 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.67@KRBTEST.COM
I20260619 15:03:27.056555 32248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.029193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:52023 (local address 127.24.103.124:39403)
0619 15:03:27.029649 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.029656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.030875 (+ 1219us) server_negotiation.cc:407] Connection header received
0619 15:03:27.035399 (+ 4524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.035405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.035503 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.035632 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:27.038233 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.039425 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.042670 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.043352 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.043455 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.048878 (+ 5423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:27.048888 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:27.048896 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:27.048942 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:27.052462 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.053479 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.053486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.053490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.053648 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.054223 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.054228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.054232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.054595 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:27.054718 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.055886 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.056141 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":223,"threads_started":1}
I20260619 15:03:27.058517 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.031452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:37219 (local address 127.24.103.126:36187)
0619 15:03:27.031935 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.031939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.036431 (+ 4492us) server_negotiation.cc:407] Connection header received
0619 15:03:27.037208 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.037218 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.037349 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.037485 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:27.039264 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.040716 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.043017 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.043709 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.043829 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.049967 (+ 6138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:27.049972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:27.049976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:27.050012 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:27.054087 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.055395 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.055401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.055405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.055571 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.056999 (+ 1428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.057004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.057007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.057356 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:27.057504 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.057951 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.058182 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":283,"threads_started":1}
I20260619 15:03:27.058944 32249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.029838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39403 (local address 127.24.103.66:52023)
0619 15:03:27.030674 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.030729 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.035096 (+ 4367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.036112 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.036125 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.036999 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:27.037984 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.038015 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.039651 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.039657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.042488 (+ 2831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.042498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.043823 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.043830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.044235 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.045256 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:27.045282 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:27.048562 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:27.052660 (+ 4098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:27.052671 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:27.052693 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:27.053328 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:27.053809 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:27.053813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:27.053817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:27.054061 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:27.054865 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:27.054876 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:27.055396 (+ 520us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.056796 (+ 1400us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.057395 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":9,"thread_start_us":362,"threads_started":1}
I20260619 15:03:27.058990 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.031339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.66:37219)
0619 15:03:27.036297 (+ 4958us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.036320 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.036446 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.037712 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.037724 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.038563 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:27.039023 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.039041 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.040954 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.040962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.042885 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.042898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.044018 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.044030 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.044244 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.045173 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:27.045210 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:27.049571 (+ 4361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:27.054490 (+ 4919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:27.054499 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:27.054505 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:27.055092 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:27.055711 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:27.055718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:27.055723 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:27.055902 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:27.057488 (+ 1586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:27.057493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:27.057663 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.057827 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.058113 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4825,"mutex_wait_us":107,"spinlock_wait_cycles":21760,"thread_start_us":4662,"threads_started":1}
I20260619 15:03:27.060632 32245 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36187
I20260619 15:03:27.061120 32245 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:27.061399 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.033893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.24.103.67:46311)
0619 15:03:27.035846 (+ 1953us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.035856 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.035885 (+ 29us) server_negotiation.cc:407] Connection header received
0619 15:03:27.035985 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.035994 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.036257 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.038645 (+ 2388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:27.038926 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.040432 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.042317 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.043182 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.043326 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.053710 (+ 10384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:27.053732 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:27.053754 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:27.053801 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:27.057367 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.058310 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.058321 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.058325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.058480 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.058968 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.058975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.058981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.059406 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:27.059678 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.060280 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.060555 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1805,"thread_start_us":1665,"threads_started":1}
I20260619 15:03:27.062101 32245 heartbeater.cc:511] Master 127.24.103.126:36187 requested a full tablet report, sending...
I20260619 15:03:27.062770 31578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:27.063125 32244 heartbeater.cc:347] Connected to a master server at 127.24.103.124:39403
I20260619 15:03:27.063251 32244 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:27.063776 32244 heartbeater.cc:511] Master 127.24.103.124:39403 requested a full tablet report, sending...
I20260619 15:03:27.064776 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.036578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:36205 (local address 127.24.103.125:44261)
0619 15:03:27.037069 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.037075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.042426 (+ 5351us) server_negotiation.cc:407] Connection header received
0619 15:03:27.042670 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.042675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.042770 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.042874 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:27.044919 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.046039 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.047820 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.048304 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.048446 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.052470 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:27.052478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:27.052482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:27.052525 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:27.059412 (+ 6887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.060275 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.060280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.060283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.060440 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.061449 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.061456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.061462 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.062860 (+ 1398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:27.063038 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.063683 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.064225 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":262,"threads_started":1}
I20260619 15:03:27.065167 31127 ts_manager.cc:194] Re-registered known tserver with Master: 638989ea9b4c4df599319b3f97e1b5d9 (127.24.103.66:34037)
I20260619 15:03:27.065716 32255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.037052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:44261 (local address 127.24.103.66:36205)
0619 15:03:27.042318 (+ 5266us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.042339 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.042529 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.043660 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.043664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.044378 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:27.044778 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.044788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.046217 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.046223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.047676 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.047689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.048498 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.048504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.048673 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.049298 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:27.049321 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:27.052286 (+ 2965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:27.059612 (+ 7326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:27.059616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:27.059620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:27.060138 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:27.060570 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:27.060594 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:27.060597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:27.061309 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:27.063115 (+ 1806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:27.063123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:27.063309 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.063912 (+ 603us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.065228 (+ 1316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5132,"thread_start_us":4941,"threads_started":1}
I20260619 15:03:27.066184 31127 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:37219
I20260619 15:03:27.067440 31925 ts_manager.cc:194] Registered new tserver with Master: 638989ea9b4c4df599319b3f97e1b5d9 (127.24.103.66:34037)
I20260619 15:03:27.067816 32246 heartbeater.cc:347] Connected to a master server at 127.24.103.125:44261
I20260619 15:03:27.067970 32246 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:27.068610 32246 heartbeater.cc:511] Master 127.24.103.125:44261 requested a full tablet report, sending...
I20260619 15:03:27.069682 31731 ts_manager.cc:194] Registered new tserver with Master: 638989ea9b4c4df599319b3f97e1b5d9 (127.24.103.66:34037)
I20260619 15:03:27.147872 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 31455
I20260619 15:03:27.164481 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:46311
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:36187
--webserver_port=42793
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.103.126:36187,127.24.103.125:44261
--tserver_master_addrs=127.24.103.126:36187,127.24.103.125:44261,127.24.103.124:39403 with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:03:27.440822 32105 heartbeater.cc:503] Master 127.24.103.126:36187 was elected leader, sending a full tablet report...
W20260619 15:03:27.500156 32257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:27.500468 32257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:27.500663 32257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:27.508539 32257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:27.508687 32257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:27.508865 32257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:03:27.519161 32257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:46311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=42793
--tserver_master_addrs=127.24.103.126:36187,127.24.103.125:44261,127.24.103.124:39403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:27.520620 32257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:27.521931 32257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:27.535734 32265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:27.535743 32263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:27.536425 32262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:27.537285 32257 server_base.cc:1034] running on GCE node
I20260619 15:03:27.537973 32257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:27.539067 32257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:27.540270 32257 hybrid_clock.cc:648] HybridClock initialized: now 1781881407540222 us; error 47 us; skew 500 ppm
I20260619 15:03:27.540622 32257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:27 dist-test-slave-zpbw krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881407, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:27.543980 32257 init.cc:377] Logged in from keytab as oryx/127.24.103.67@KRBTEST.COM (short username oryx)
I20260619 15:03:27.545482 32257 webserver.cc:460] Webserver started at http://127.24.103.67:42793/ using document root <none> and password file <none>
I20260619 15:03:27.546216 32257 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:27.546378 32257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:27.552008 32257 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 15:03:27.554881 32272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:27.556385 32257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 15:03:27.556581 32257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "8f29062c40dd49a8952f4d893bec9fb5"
format_stamp: "Formatted at 2026-06-19 15:03:24 on dist-test-slave-zpbw"
server_key: "0ed15d5f2d833f16ae51be1f137b720b"
server_key_iv: "1a11c1c72e8abf4bd0ee19f4f5132d18"
server_key_version: "encryptionkey@0"
I20260619 15:03:27.557778 32257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:27.577898 32257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:27.581954 32257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:27.582191 32257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:27.583182 32257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:27.584528 32257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:27.584623 32257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:27.584707 32257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:27.584764 32257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:27.618062 32257 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:46311
I20260619 15:03:27.618145 32384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:46311 every 8 connection(s)
I20260619 15:03:27.619606 32257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:03:27.629595 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 32257
Jun 19 15:03:27 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881407, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
Jun 19 15:03:27 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881407, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.124@KRBTEST.COM
I20260619 15:03:27.668040 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.623698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.67:47835)
0619 15:03:27.624498 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.624553 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.627489 (+ 2936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.648385 (+ 20896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.648389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.648830 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:27.649065 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.649088 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.650730 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.650736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.652230 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.652240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.653323 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.653332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.653597 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.654343 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:27.654366 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:27.657384 (+ 3018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:27.662029 (+ 4645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:27.662041 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:27.662063 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:27.662747 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:27.663269 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:27.663279 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:27.663285 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:27.663559 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:27.664398 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:27.664408 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:27.664958 (+ 550us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.665279 (+ 321us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.665969 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":401,"threads_started":1}
I20260619 15:03:27.668809 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.625549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:60381 (local address 127.24.103.124:39403)
0619 15:03:27.644140 (+ 18591us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.644150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.645061 (+ 911us) server_negotiation.cc:407] Connection header received
0619 15:03:27.645344 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.645350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.645433 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.645549 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:27.648408 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.649564 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.652198 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.652837 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.652935 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.661161 (+ 8226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:27.661169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:27.661174 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:27.661219 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:27.665128 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.666129 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.666135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.666139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.666299 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.666826 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.666831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.666835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.667198 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:27.667349 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.668034 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.668269 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18398,"thread_start_us":18620,"threads_started":1}
I20260619 15:03:27.669792 32399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.622766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:47835 (local address 127.24.103.126:36187)
0619 15:03:27.648020 (+ 25254us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.648027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.648047 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:03:27.648119 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.648124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.648201 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.649228 (+ 1027us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:27.649428 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.650564 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.652413 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.653063 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.653277 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.657675 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:27.657682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:27.657687 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:27.657730 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:27.661609 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.662955 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.662962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.662965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.663131 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.663739 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.663745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.663748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.664204 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:27.668811 (+ 4607us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.669080 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.669335 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25086,"thread_start_us":33306,"threads_started":1}
I20260619 15:03:27.669883 32391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.625176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39403 (local address 127.24.103.67:60381)
0619 15:03:27.644862 (+ 19686us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.644896 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.645127 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.646148 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.646163 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.647299 (+ 1136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:27.648196 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.648222 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.649790 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.649795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.652021 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.652035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.653986 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.653995 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.655730 (+ 1735us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.656552 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:27.656584 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:27.660673 (+ 4089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:27.665302 (+ 4629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:27.665307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:27.665311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:27.665969 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:27.666474 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:27.666480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:27.666484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:27.666644 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:27.667431 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:27.667438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:27.667605 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.667749 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.669275 (+ 1526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19502,"thread_start_us":20625,"threads_started":1}
I20260619 15:03:27.674145 32385 heartbeater.cc:347] Connected to a master server at 127.24.103.124:39403
I20260619 15:03:27.674448 32385 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:27.675588 32385 heartbeater.cc:511] Master 127.24.103.124:39403 requested a full tablet report, sending...
I20260619 15:03:27.677743 31925 ts_manager.cc:194] Registered new tserver with Master: 8f29062c40dd49a8952f4d893bec9fb5 (127.24.103.67:46311)
I20260619 15:03:27.678234 32386 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36187
I20260619 15:03:27.678362 32386 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:27.678895 32386 heartbeater.cc:511] Master 127.24.103.126:36187 requested a full tablet report, sending...
I20260619 15:03:27.679966 31127 ts_manager.cc:194] Re-registered known tserver with Master: 8f29062c40dd49a8952f4d893bec9fb5 (127.24.103.67:46311)
Jun 19 15:03:27 dist-test-slave-zpbw krb5kdc[31082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881407, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
I20260619 15:03:27.681149 31127 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:47835
I20260619 15:03:27.689838 32398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.646543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:34373 (local address 127.24.103.125:44261)
0619 15:03:27.671260 (+ 24717us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.671267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.671286 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:03:27.671360 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.671367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.671476 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.671614 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:27.672412 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.673468 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.675611 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.676243 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.676357 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.681067 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:27.681074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:27.681078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:27.681127 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:27.685443 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.686687 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.686692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.686696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.686916 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.687412 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.687419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.687423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.687833 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:27.687966 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.689132 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.689372 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12728,"thread_start_us":20795,"threads_started":1}
I20260619 15:03:27.690295 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.646450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:44261 (local address 127.24.103.67:34373)
0619 15:03:27.668496 (+ 22046us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.668517 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.668624 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.671652 (+ 3028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.671655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.672001 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:27.672230 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.672243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.673732 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.673740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.675428 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.675446 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.676565 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.676592 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.676772 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.677431 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:27.677454 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:27.680858 (+ 3404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:27.685957 (+ 5099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:27.685964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:27.685970 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:27.686553 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:27.687063 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:27.687070 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:27.687076 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:27.687242 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:27.688043 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:27.688048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:27.688221 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.688362 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.688743 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21931,"thread_start_us":25790,"threads_started":1}
I20260619 15:03:27.691593 32387 heartbeater.cc:347] Connected to a master server at 127.24.103.125:44261
I20260619 15:03:27.691740 32387 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:27.692262 32387 heartbeater.cc:511] Master 127.24.103.125:44261 requested a full tablet report, sending...
I20260619 15:03:27.693532 31731 ts_manager.cc:194] Registered new tserver with Master: 8f29062c40dd49a8952f4d893bec9fb5 (127.24.103.67:46311)
I20260619 15:03:27.695189 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.647493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.24.103.126:36187)
0619 15:03:27.656403 (+ 8910us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.656410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.660815 (+ 4405us) server_negotiation.cc:407] Connection header received
0619 15:03:27.672111 (+ 11296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.672116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.672192 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.672303 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:27.675338 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.676515 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.679648 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.680267 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.680388 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.685528 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:27.685537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:27.685541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:27.685586 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:27.690035 (+ 4449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.691415 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.691422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.691425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.691603 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.692783 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.692788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.692792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.693154 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:27.693411 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.694406 (+ 995us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.694685 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":8763,"thread_start_us":278,"threads_started":1}
I20260619 15:03:27.697405 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.660243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.24.103.125:44261)
0619 15:03:27.671137 (+ 10894us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.671146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.674582 (+ 3436us) server_negotiation.cc:407] Connection header received
0619 15:03:27.674884 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.674889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.674960 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.675058 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:27.677624 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.679435 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.682626 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.683473 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.683580 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.688521 (+ 4941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:27.688527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:27.688532 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:27.688566 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:27.693050 (+ 4484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.693851 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.693856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.693859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.694024 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.694630 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.694637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.694640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.694986 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:27.695253 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.696295 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.696564 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10714,"thread_start_us":10907,"threads_started":1}
I20260619 15:03:27.703676 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.677335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.24.103.124:39403)
0619 15:03:27.677511 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.677523 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.684780 (+ 7257us) server_negotiation.cc:407] Connection header received
0619 15:03:27.685103 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.685112 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.685197 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.685272 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:27.689196 (+ 3924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.690314 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.692192 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.692927 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.693582 (+ 655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.697900 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:27.697905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:27.697909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:27.697944 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:27.700829 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.701550 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.701563 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.701566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.701694 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:27.702085 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:27.702089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:27.702091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:27.702372 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:27.702604 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.703052 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.703286 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:03:27.700060 31127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:03:27.708567 31127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:03:27.749703 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.727797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:46311 (local address 127.0.0.1:34838)
0619 15:03:27.740760 (+ 12963us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.740776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.740928 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.742373 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.742377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.742401 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:27.743002 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.743017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.745218 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.745225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.747316 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.747346 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.748668 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.748674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.748925 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.748965 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.749065 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.749232 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12840,"mutex_wait_us":28,"thread_start_us":12706,"threads_started":1}
I20260619 15:03:27.749871 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.727072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38491 (local address 127.0.0.1:41436)
0619 15:03:27.740042 (+ 12970us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.740069 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.740237 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.742049 (+ 1812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.742057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.742101 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:27.742703 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.742719 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.744603 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.744611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.747182 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.747193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.749072 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.749079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.749227 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.749256 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.749340 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.749475 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12797,"mutex_wait_us":40,"thread_start_us":10415,"threads_started":1}
I20260619 15:03:27.749943 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.728405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.24.103.65:38491)
0619 15:03:27.741429 (+ 13024us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.741437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.741467 (+ 30us) server_negotiation.cc:407] Connection header received
0619 15:03:27.741560 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.741571 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.741799 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.741994 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:27.743075 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.744082 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.747382 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.748880 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.749014 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.749196 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.749439 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.749633 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12846,"thread_start_us":12747,"threads_started":1}
I20260619 15:03:27.750453 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.729138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34838 (local address 127.24.103.67:46311)
0619 15:03:27.737278 (+ 8140us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.737292 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.740915 (+ 3623us) server_negotiation.cc:407] Connection header received
0619 15:03:27.741080 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.741091 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.741459 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.741684 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:27.743964 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.745087 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.747488 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.748441 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.748632 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.748894 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.750101 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.750233 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7920,"thread_start_us":7743,"threads_started":1}
I20260619 15:03:27.754024 32039 tablet_service.cc:1467] Processing CreateTablet for tablet 3541425ee23c4d5f9336a7115b5fc4db (DEFAULT_TABLE table=test-table [id=00de3488d0dc4bdea664f048720d2020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:27.754024 32320 tablet_service.cc:1467] Processing CreateTablet for tablet 3541425ee23c4d5f9336a7115b5fc4db (DEFAULT_TABLE table=test-table [id=00de3488d0dc4bdea664f048720d2020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:27.755728 32320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3541425ee23c4d5f9336a7115b5fc4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:27.756042 32039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3541425ee23c4d5f9336a7115b5fc4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:27.756150 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.726946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.24.103.66:34037)
0619 15:03:27.743226 (+ 16280us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.743234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.743255 (+ 21us) server_negotiation.cc:407] Connection header received
0619 15:03:27.743333 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.743338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.743510 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.743715 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:27.745284 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.747137 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.750714 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.751830 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.751949 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.752074 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.755683 (+ 3609us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.755864 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16118,"thread_start_us":16008,"threads_started":1}
I20260619 15:03:27.756529 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.726660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34037 (local address 127.0.0.1:56114)
0619 15:03:27.737762 (+ 11102us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.737783 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.737929 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.743702 (+ 5773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.743706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.743733 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:27.744104 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.744117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.747270 (+ 3153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.747277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.749442 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.749457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.754954 (+ 5497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.754961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.755424 (+ 463us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.755460 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.755678 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.755955 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10967,"mutex_wait_us":31,"thread_start_us":15519,"threads_started":1}
I20260619 15:03:27.759768 32179 tablet_service.cc:1467] Processing CreateTablet for tablet 3541425ee23c4d5f9336a7115b5fc4db (DEFAULT_TABLE table=test-table [id=00de3488d0dc4bdea664f048720d2020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:27.761036 32179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3541425ee23c4d5f9336a7115b5fc4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:27.772737 32420 tablet_bootstrap.cc:492] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5: Bootstrap starting.
I20260619 15:03:27.777124 32420 tablet_bootstrap.cc:654] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:27.777432 32422 tablet_bootstrap.cc:492] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9: Bootstrap starting.
I20260619 15:03:27.780531 32420 log.cc:824] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:27.781884 32422 tablet_bootstrap.cc:654] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:27.781888 32421 tablet_bootstrap.cc:492] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102: Bootstrap starting.
I20260619 15:03:27.783296 32422 log.cc:824] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:27.785027 32420 tablet_bootstrap.cc:492] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5: No bootstrap required, opened a new log
I20260619 15:03:27.785316 32420 ts_tablet_manager.cc:1397] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260619 15:03:27.785768 32421 tablet_bootstrap.cc:654] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:27.786523 32422 tablet_bootstrap.cc:492] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9: No bootstrap required, opened a new log
I20260619 15:03:27.786885 32422 ts_tablet_manager.cc:1397] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20260619 15:03:27.787160 32421 log.cc:824] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:27.792125 32421 tablet_bootstrap.cc:492] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102: No bootstrap required, opened a new log
I20260619 15:03:27.792426 32421 ts_tablet_manager.cc:1397] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260619 15:03:27.794021 32420 raft_consensus.cc:348] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:27.794399 32420 raft_consensus.cc:374] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:27.794486 32420 raft_consensus.cc:729] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f29062c40dd49a8952f4d893bec9fb5, State: Initialized, Role: FOLLOWER
I20260619 15:03:27.795543 32420 consensus_queue.cc:260] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:27.797320 32422 raft_consensus.cc:348] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:27.797766 32422 raft_consensus.cc:374] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:27.797880 32422 raft_consensus.cc:729] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638989ea9b4c4df599319b3f97e1b5d9, State: Initialized, Role: FOLLOWER
I20260619 15:03:27.798588 32422 consensus_queue.cc:260] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:27.799296 32420 ts_tablet_manager.cc:1428] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260619 15:03:27.799783 32386 heartbeater.cc:503] Master 127.24.103.126:36187 was elected leader, sending a full tablet report...
I20260619 15:03:27.800921 32245 heartbeater.cc:503] Master 127.24.103.126:36187 was elected leader, sending a full tablet report...
I20260619 15:03:27.805896 32422 ts_tablet_manager.cc:1428] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9: Time spent starting tablet: real 0.019s user 0.010s sys 0.004s
I20260619 15:03:27.805850 32421 raft_consensus.cc:348] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:27.806624 32421 raft_consensus.cc:374] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:27.806780 32421 raft_consensus.cc:729] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c90e3825b54f8f91d82c5ba5e9c102, State: Initialized, Role: FOLLOWER
I20260619 15:03:27.808015 32421 consensus_queue.cc:260] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:27.810410 32421 ts_tablet_manager.cc:1428] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260619 15:03:27.836056 32428 raft_consensus.cc:482] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:03:27.836510 32428 raft_consensus.cc:504] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:27.836668 32427 raft_consensus.cc:482] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:03:27.836956 32427 raft_consensus.cc:504] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:27.838507 32427 leader_election.cc:288] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c90e3825b54f8f91d82c5ba5e9c102 (127.24.103.65:38491), 8f29062c40dd49a8952f4d893bec9fb5 (127.24.103.67:46311)
I20260619 15:03:27.840591 32428 leader_election.cc:288] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 638989ea9b4c4df599319b3f97e1b5d9 (127.24.103.66:34037), 8f29062c40dd49a8952f4d893bec9fb5 (127.24.103.67:46311)
I20260619 15:03:27.851306 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.839943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:54171 (local address 127.24.103.65:38491)
0619 15:03:27.840158 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.840163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.841308 (+ 1145us) server_negotiation.cc:407] Connection header received
0619 15:03:27.841652 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.841661 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.841757 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.841829 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:27.842806 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.844364 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.847876 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.849633 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.849807 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.849929 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.850718 (+ 789us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.850937 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 15:03:27.852275 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.838694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38491 (local address 127.24.103.66:54171)
0619 15:03:27.841179 (+ 2485us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.841216 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.841452 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.842063 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.842071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.842104 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:27.842581 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.842595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.844629 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.844637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.847623 (+ 2986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.847649 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.850183 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.850192 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.850479 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.850531 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.851602 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.851810 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2291,"thread_start_us":2628,"threads_started":1}
I20260619 15:03:27.853286 32059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3541425ee23c4d5f9336a7115b5fc4db" candidate_uuid: "638989ea9b4c4df599319b3f97e1b5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" is_pre_election: true
I20260619 15:03:27.853783 32059 raft_consensus.cc:2454] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 638989ea9b4c4df599319b3f97e1b5d9 in term 0.
I20260619 15:03:27.854509 32132 leader_election.cc:302] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 638989ea9b4c4df599319b3f97e1b5d9, 76c90e3825b54f8f91d82c5ba5e9c102; no voters:
I20260619 15:03:27.854960 32427 raft_consensus.cc:2781] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:03:27.855109 32427 raft_consensus.cc:482] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:03:27.855264 32427 raft_consensus.cc:3037] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:27.859172 32427 raft_consensus.cc:504] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:27.860278 32059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3541425ee23c4d5f9336a7115b5fc4db" candidate_uuid: "638989ea9b4c4df599319b3f97e1b5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c90e3825b54f8f91d82c5ba5e9c102"
I20260619 15:03:27.860513 32059 raft_consensus.cc:3037] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:27.861409 32427 leader_election.cc:288] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [CANDIDATE]: Term 1 election: Requested vote from peers 76c90e3825b54f8f91d82c5ba5e9c102 (127.24.103.65:38491), 8f29062c40dd49a8952f4d893bec9fb5 (127.24.103.67:46311)
I20260619 15:03:27.862454 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.839325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:42921 (local address 127.24.103.67:46311)
0619 15:03:27.840009 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.840017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.853174 (+ 13157us) server_negotiation.cc:407] Connection header received
0619 15:03:27.853515 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.853523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.853609 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.853681 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:27.854379 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.855774 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.858767 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.860163 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.860354 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.860446 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.861949 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.862114 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522}
I20260619 15:03:27.863143 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.839247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:46311 (local address 127.24.103.66:42921)
0619 15:03:27.853067 (+ 13820us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.853092 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.853299 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.853925 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.853932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.853957 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:27.854187 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.854201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.856016 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.856027 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.858587 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.858600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.860693 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.860713 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.861472 (+ 759us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.861579 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.861682 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.862760 (+ 1078us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13686,"thread_start_us":18586,"threads_started":1}
I20260619 15:03:27.864763 32340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3541425ee23c4d5f9336a7115b5fc4db" candidate_uuid: "638989ea9b4c4df599319b3f97e1b5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f29062c40dd49a8952f4d893bec9fb5" is_pre_election: true
I20260619 15:03:27.865346 32340 raft_consensus.cc:2454] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 638989ea9b4c4df599319b3f97e1b5d9 in term 0.
I20260619 15:03:27.864785 32339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3541425ee23c4d5f9336a7115b5fc4db" candidate_uuid: "638989ea9b4c4df599319b3f97e1b5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f29062c40dd49a8952f4d893bec9fb5"
I20260619 15:03:27.866060 32339 raft_consensus.cc:3037] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:27.866438 32059 raft_consensus.cc:2454] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 638989ea9b4c4df599319b3f97e1b5d9 in term 1.
I20260619 15:03:27.868034 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.840152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:59861 (local address 127.24.103.67:46311)
0619 15:03:27.852786 (+ 12634us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.852793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.852816 (+ 23us) server_negotiation.cc:407] Connection header received
0619 15:03:27.855077 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.855085 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.855214 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.855963 (+ 749us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:27.856347 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.857605 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.862512 (+ 4907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.863769 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.863947 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.864014 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.867436 (+ 3422us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.867605 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12493,"thread_start_us":14783,"threads_started":1}
I20260619 15:03:27.868253 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.838764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:46311 (local address 127.24.103.65:59861)
0619 15:03:27.839814 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.839831 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.839914 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.855393 (+ 15479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.855400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.855437 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:27.855784 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.855796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.858148 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.858154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.862317 (+ 4163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.862336 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.864981 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.864989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.867243 (+ 2254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.867285 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.867374 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.867809 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":1916,"thread_start_us":763,"threads_started":1}
I20260619 15:03:27.869138 32132 leader_election.cc:302] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 638989ea9b4c4df599319b3f97e1b5d9, 76c90e3825b54f8f91d82c5ba5e9c102; no voters:
I20260619 15:03:27.869405 32339 raft_consensus.cc:2454] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 638989ea9b4c4df599319b3f97e1b5d9 in term 1.
I20260619 15:03:27.869648 32427 raft_consensus.cc:2781] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:27.869853 32340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3541425ee23c4d5f9336a7115b5fc4db" candidate_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f29062c40dd49a8952f4d893bec9fb5" is_pre_election: true
I20260619 15:03:27.870172 32340 raft_consensus.cc:2379] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76c90e3825b54f8f91d82c5ba5e9c102 in current term 1: Already voted for candidate 638989ea9b4c4df599319b3f97e1b5d9 in this term.
I20260619 15:03:27.870621 32427 raft_consensus.cc:686] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 1 LEADER]: Becoming Leader. State: Replica: 638989ea9b4c4df599319b3f97e1b5d9, State: Running, Role: LEADER
I20260619 15:03:27.871129 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.841616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:44179 (local address 127.24.103.66:34037)
0619 15:03:27.841770 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.841775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.841790 (+ 15us) server_negotiation.cc:407] Connection header received
0619 15:03:27.841841 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.841846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.841903 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.841985 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:27.859500 (+ 17515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.860466 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.868117 (+ 7651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.869919 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.870117 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.870206 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.870705 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.870865 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 15:03:27.871275 32427 consensus_queue.cc:237] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:27.871973 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.838617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34037 (local address 127.24.103.65:44179)
0619 15:03:27.839662 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:27.839689 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:27.839900 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:27.858887 (+ 18987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:27.858895 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:27.858930 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:27.859341 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.859354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.860636 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.860640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.867027 (+ 6387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:27.867042 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.870163 (+ 3121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.870170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:27.870447 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.870500 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 15:03:27.871472 (+ 972us) client_negotiation.cc:240] Negotiation successful
0619 15:03:27.871668 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":2,"thread_start_us":447,"threads_started":1}
I20260619 15:03:27.872890 32199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3541425ee23c4d5f9336a7115b5fc4db" candidate_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638989ea9b4c4df599319b3f97e1b5d9" is_pre_election: true
W20260619 15:03:27.873917 32388 tablet.cc:2369] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:03:27.873988 31995 leader_election.cc:302] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76c90e3825b54f8f91d82c5ba5e9c102; no voters: 638989ea9b4c4df599319b3f97e1b5d9, 8f29062c40dd49a8952f4d893bec9fb5
I20260619 15:03:27.874641 32428 raft_consensus.cc:2726] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 15:03:27.877163 31125 catalog_manager.cc:5495] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 638989ea9b4c4df599319b3f97e1b5d9 (127.24.103.66). New cstate: current_term: 1 leader_uuid: "638989ea9b4c4df599319b3f97e1b5d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } health_report { overall_health: UNKNOWN } } }
W20260619 15:03:27.886898 32107 tablet.cc:2369] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:03:27.908358 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.902929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.24.103.66:34037)
0619 15:03:27.903102 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.903107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.903146 (+ 39us) server_negotiation.cc:407] Connection header received
0619 15:03:27.903292 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.903295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.903372 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.903567 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:03:27.903922 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.904744 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.906119 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.906508 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.906612 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.907103 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:03:27.907725 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:03:27.907829 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:27.908007 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 15:03:27.908108 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 15:03:27.916894 32340 raft_consensus.cc:1261] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5 [term 1 FOLLOWER]: Refusing update from remote peer 638989ea9b4c4df599319b3f97e1b5d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:27.916931 32059 raft_consensus.cc:1261] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [term 1 FOLLOWER]: Refusing update from remote peer 638989ea9b4c4df599319b3f97e1b5d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:27.917845 32434 consensus_queue.cc:1035] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:27.918254 32427 consensus_queue.cc:1035] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:27.930131 32438 mvcc.cc:205] Tried to move back new op lower bound from 7298586246817017856 to 7298586246646177792. Current Snapshot: MvccSnapshot[applied={T|T < 7298586246817017856}]
I20260619 15:03:27.939064 32439 mvcc.cc:205] Tried to move back new op lower bound from 7298586246817017856 to 7298586246646177792. Current Snapshot: MvccSnapshot[applied={T|T < 7298586246817017856}]
I20260619 15:03:27.939530 32440 mvcc.cc:205] Tried to move back new op lower bound from 7298586246817017856 to 7298586246646177792. Current Snapshot: MvccSnapshot[applied={T|T < 7298586246817017856}]
I20260619 15:03:27.953406 31126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:27.953744 31126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:27.961135 31126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e27d27671e74cb798fe7148e4eb0ce8: Sending DeleteTablet for 3 replicas of tablet 3541425ee23c4d5f9336a7115b5fc4db
I20260619 15:03:27.962675 32039 tablet_service.cc:1514] Processing DeleteTablet for tablet 3541425ee23c4d5f9336a7115b5fc4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:27 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:41436
I20260619 15:03:27.962934 32179 tablet_service.cc:1514] Processing DeleteTablet for tablet 3541425ee23c4d5f9336a7115b5fc4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:27 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:56114
I20260619 15:03:27.963954 32320 tablet_service.cc:1514] Processing DeleteTablet for tablet 3541425ee23c4d5f9336a7115b5fc4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:27 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:34838
I20260619 15:03:27.974515 32448 tablet_replica.cc:330] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102: stopping tablet replica
I20260619 15:03:27.974577 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 31095
I20260619 15:03:27.975108 32448 raft_consensus.cc:2229] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:27.976112 32448 raft_consensus.cc:2258] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:27.976131 31157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:27.977020 32450 tablet_replica.cc:330] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5: stopping tablet replica
I20260619 15:03:27.977216 32449 tablet_replica.cc:330] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9: stopping tablet replica
I20260619 15:03:27.977483 32450 raft_consensus.cc:2229] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:27.977985 32450 raft_consensus.cc:2258] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:27.978202 32449 raft_consensus.cc:2229] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:03:27.979897 32449 raft_consensus.cc:2258] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:27.982687 32450 ts_tablet_manager.cc:1905] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:27.980919 32448 ts_tablet_manager.cc:1905] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:27.984937 32449 ts_tablet_manager.cc:1905] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:27.992820 32449 ts_tablet_manager.cc:1918] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:27.993109 32449 log.cc:1196] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/3541425ee23c4d5f9336a7115b5fc4db
I20260619 15:03:27.993862 32449 ts_tablet_manager.cc:1939] T 3541425ee23c4d5f9336a7115b5fc4db P 638989ea9b4c4df599319b3f97e1b5d9: Deleting consensus metadata
I20260619 15:03:27.996901 32448 ts_tablet_manager.cc:1918] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:27.997118 32448 log.cc:1196] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/3541425ee23c4d5f9336a7115b5fc4db
I20260619 15:03:27.998133 32450 ts_tablet_manager.cc:1918] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:27.998322 32450 log.cc:1196] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/3541425ee23c4d5f9336a7115b5fc4db
I20260619 15:03:27.999287 32448 ts_tablet_manager.cc:1939] T 3541425ee23c4d5f9336a7115b5fc4db P 76c90e3825b54f8f91d82c5ba5e9c102: Deleting consensus metadata
I20260619 15:03:28.001068 32450 ts_tablet_manager.cc:1939] T 3541425ee23c4d5f9336a7115b5fc4db P 8f29062c40dd49a8952f4d893bec9fb5: Deleting consensus metadata
I20260619 15:03:28.004786 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.965689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.24.103.125:44261)
0619 15:03:27.965870 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.965877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:27.973154 (+ 7277us) server_negotiation.cc:407] Connection header received
0619 15:03:27.973408 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:27.973414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:27.973523 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:27.973658 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:27.981462 (+ 7804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.982602 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.984747 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:27.985439 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:27.985572 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:27.994817 (+ 9245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:27.994826 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:27.994832 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:27.994881 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:27.999175 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:28.002060 (+ 2885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:28.002067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:28.002071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:28.002237 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:28.002737 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:28.002742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:28.002746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:28.003184 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:28.003439 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:28.004083 (+ 644us) server_negotiation.cc:299] Negotiation successful
0619 15:03:28.004430 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 15:03:28.023692 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:27.972929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.24.103.124:39403)
0619 15:03:27.973128 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:27.973135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:28.004901 (+ 31766us) server_negotiation.cc:407] Connection header received
0619 15:03:28.005540 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:28.005548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:28.005646 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:28.005726 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:28.007914 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:28.008943 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:28.011325 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:28.011990 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:28.012149 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:28.016140 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:28.016146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:28.016152 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:28.016204 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:28.020054 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:28.020956 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:28.020963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:28.020968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:28.021134 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:28.021566 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:28.021571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:28.021574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:28.021966 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:28.022253 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:28.022608 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 15:03:28.022883 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 15:03:28.140710 31113 catalog_manager.cc:4841] TS 8f29062c40dd49a8952f4d893bec9fb5 (127.24.103.67:46311): tablet 3541425ee23c4d5f9336a7115b5fc4db (table test-table [id=00de3488d0dc4bdea664f048720d2020]) successfully deleted
I20260619 15:03:28.141100 31111 catalog_manager.cc:4841] TS 76c90e3825b54f8f91d82c5ba5e9c102 (127.24.103.65:38491): tablet 3541425ee23c4d5f9336a7115b5fc4db (table test-table [id=00de3488d0dc4bdea664f048720d2020]) successfully deleted
I20260619 15:03:28.141185 31114 catalog_manager.cc:4841] TS 638989ea9b4c4df599319b3f97e1b5d9 (127.24.103.66:34037): tablet 3541425ee23c4d5f9336a7115b5fc4db (table test-table [id=00de3488d0dc4bdea664f048720d2020]) successfully deleted
I20260619 15:03:28.141335 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 31095
I20260619 15:03:28.192517 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:28.170123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.24.103.125:44261)
0619 15:03:28.170444 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:28.170453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:28.170885 (+ 432us) server_negotiation.cc:407] Connection header received
0619 15:03:28.171207 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:28.171213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:28.171293 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:28.171375 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:28.174721 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:28.176305 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:28.179129 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:28.180123 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:28.180271 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:28.185499 (+ 5228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:28.185509 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:28.185518 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:28.185583 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:28.189299 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:28.190141 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:28.190147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:28.190151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:28.190363 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:28.190867 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:28.190873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:28.190877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:28.191236 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:28.191476 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:28.191936 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 15:03:28.192158 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260619 15:03:28.203179 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:28.171070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.24.103.124:39403)
0619 15:03:28.171283 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:28.171289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:28.187535 (+ 16246us) server_negotiation.cc:407] Connection header received
0619 15:03:28.187695 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:28.187701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:28.187796 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:28.187916 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:28.190056 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:28.191262 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:28.193238 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:28.193917 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:28.194043 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:28.197591 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:28.197597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:28.197600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:28.197639 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:28.200354 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:28.201236 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:28.201240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:28.201243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:28.201359 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:28.201769 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:28.201772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:28.201774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:28.202057 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:28.202228 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:28.202645 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 15:03:28.202896 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 15:03:28.940316 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:28.939508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.65:44793)
0619 15:03:28.939954 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:28.940109 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":218,"threads_started":1}
W20260619 15:03:28.940800 32105 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:36187 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
I20260619 15:03:28.948817 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:28.947573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.67:33597)
0619 15:03:28.948454 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:28.948635 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":602,"threads_started":1}
W20260619 15:03:28.949194 32386 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:36187 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
I20260619 15:03:28.951946 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:28.951157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.66:37001)
0619 15:03:28.951594 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:28.951714 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":248,"threads_started":1}
W20260619 15:03:28.952342 32245 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:36187 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
I20260619 15:03:29.461723 32462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0e27d27671e74cb798fe7148e4eb0ce8)
I20260619 15:03:29.462011 32462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } }
I20260619 15:03:29.462673 32463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0e27d27671e74cb798fe7148e4eb0ce8)
I20260619 15:03:29.462989 32463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } }
I20260619 15:03:29.463797 32462 leader_election.cc:288] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187), 45ddcfea2edc41979fcb0a55eb3662d7 (127.24.103.124:39403)
I20260619 15:03:29.464228 32463 leader_election.cc:288] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187), 8fe5da68b9e34c32aa5758e08a0bdd53 (127.24.103.125:44261)
I20260619 15:03:29.465444 32465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.464396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60362)
0619 15:03:29.464950 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.465110 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":272,"threads_started":1}
I20260619 15:03:29.465234 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.463921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60348)
0619 15:03:29.464780 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.464957 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":549,"threads_started":1}
W20260619 15:03:29.473471 31911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
W20260619 15:03:29.473663 31718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
I20260619 15:03:29.479810 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.464953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39403 (local address 127.0.0.1:35860)
0619 15:03:29.471344 (+ 6391us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.471378 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:29.471598 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:29.473042 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:29.473049 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:29.473120 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:29.473420 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:29.473435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:29.474817 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:29.474825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:29.477014 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:29.477027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:29.478899 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:29.478907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:29.479145 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:29.479183 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 15:03:29.479288 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 15:03:29.479439 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6222,"mutex_wait_us":3,"thread_start_us":8405,"threads_started":1}
I20260619 15:03:29.480918 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.465400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.24.103.124:39403)
0619 15:03:29.472440 (+ 7040us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:29.472447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:29.472467 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:03:29.472532 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:29.472540 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:29.472605 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:29.472689 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:29.473614 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:29.474662 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:29.477199 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:29.478502 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:29.478711 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:29.480300 (+ 1589us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:29.480362 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 15:03:29.480511 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6922,"thread_start_us":6797,"threads_started":1}
I20260619 15:03:29.481289 32465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.465060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:44261 (local address 127.0.0.1:53742)
0619 15:03:29.465826 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.465849 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:29.466024 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:29.472564 (+ 6540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:29.472579 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:29.472600 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:29.472871 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:29.472889 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:29.476545 (+ 3656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:29.476551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:29.478718 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:29.478730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:29.480453 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:29.480464 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:29.480677 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:29.480724 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 15:03:29.480797 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 15:03:29.480943 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":1,"thread_start_us":7968,"threads_started":1}
I20260619 15:03:29.482019 32468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.465360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.24.103.125:44261)
0619 15:03:29.472238 (+ 6878us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:29.472246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:29.472267 (+ 21us) server_negotiation.cc:407] Connection header received
0619 15:03:29.472335 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:29.472340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:29.472418 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:29.474850 (+ 2432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:29.475196 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:29.476379 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:29.478934 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:29.480249 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:29.481575 (+ 1326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:29.481628 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:29.481680 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 15:03:29.481769 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6757,"thread_start_us":6528,"threads_started":1}
I20260619 15:03:29.482223 32465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.481375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60364)
0619 15:03:29.481843 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.481976 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":49}
W20260619 15:03:29.482476 31911 leader_election.cc:334] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187): Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
I20260619 15:03:29.482802 31741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" is_pre_election: true
I20260619 15:03:29.483192 31741 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45ddcfea2edc41979fcb0a55eb3662d7 in term 1.
I20260619 15:03:29.483402 31935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" is_pre_election: true
I20260619 15:03:29.483752 31935 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fe5da68b9e34c32aa5758e08a0bdd53 in term 1.
I20260619 15:03:29.483776 31911 leader_election.cc:302] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45ddcfea2edc41979fcb0a55eb3662d7, 8fe5da68b9e34c32aa5758e08a0bdd53; no voters: 0e27d27671e74cb798fe7148e4eb0ce8
I20260619 15:03:29.484120 32463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 15:03:29.484236 32463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0e27d27671e74cb798fe7148e4eb0ce8)
I20260619 15:03:29.484335 32463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 1 FOLLOWER]: Advancing to term 2
I20260619 15:03:29.484308 31718 leader_election.cc:302] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45ddcfea2edc41979fcb0a55eb3662d7, 8fe5da68b9e34c32aa5758e08a0bdd53; no voters:
I20260619 15:03:29.484467 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.483818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60378)
0619 15:03:29.484067 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.484160 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 15:03:29.484688 32462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260619 15:03:29.484694 31718 leader_election.cc:334] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187): Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
I20260619 15:03:29.484826 32462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0e27d27671e74cb798fe7148e4eb0ce8)
I20260619 15:03:29.484946 32462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 1 FOLLOWER]: Advancing to term 2
I20260619 15:03:29.487547 32463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } }
I20260619 15:03:29.487998 32462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } }
I20260619 15:03:29.488363 32463 leader_election.cc:288] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [CANDIDATE]: Term 2 election: Requested vote from peers 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187), 8fe5da68b9e34c32aa5758e08a0bdd53 (127.24.103.125:44261)
I20260619 15:03:29.488767 32462 leader_election.cc:288] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [CANDIDATE]: Term 2 election: Requested vote from peers 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187), 45ddcfea2edc41979fcb0a55eb3662d7 (127.24.103.124:39403)
I20260619 15:03:29.489174 31741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53"
I20260619 15:03:29.489439 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.488845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60406)
0619 15:03:29.489222 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.489303 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242}
I20260619 15:03:29.489470 31741 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 45ddcfea2edc41979fcb0a55eb3662d7 in current term 2: Already voted for candidate 8fe5da68b9e34c32aa5758e08a0bdd53 in this term.
I20260619 15:03:29.489362 31935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "45ddcfea2edc41979fcb0a55eb3662d7"
I20260619 15:03:29.489675 31935 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fe5da68b9e34c32aa5758e08a0bdd53 in current term 2: Already voted for candidate 45ddcfea2edc41979fcb0a55eb3662d7 in this term.
I20260619 15:03:29.489673 32465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.488536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60390)
0619 15:03:29.488723 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.488819 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 15:03:29.490768 32465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.490370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60416)
0619 15:03:29.490555 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.490648 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 15:03:29.490852 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.490478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60420)
0619 15:03:29.490626 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.490689 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 15:03:29.491053 31718 leader_election.cc:334] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187): Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
W20260619 15:03:29.491060 31911 leader_election.cc:334] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187): Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
I20260619 15:03:29.491194 31718 leader_election.cc:302] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fe5da68b9e34c32aa5758e08a0bdd53; no voters: 0e27d27671e74cb798fe7148e4eb0ce8, 45ddcfea2edc41979fcb0a55eb3662d7
I20260619 15:03:29.491225 31911 leader_election.cc:302] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45ddcfea2edc41979fcb0a55eb3662d7; no voters: 0e27d27671e74cb798fe7148e4eb0ce8, 8fe5da68b9e34c32aa5758e08a0bdd53
I20260619 15:03:29.491475 32462 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 15:03:29.491503 32463 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 15:03:29.942348 32472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.941602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.65:52329)
0619 15:03:29.942010 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.942143 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":205,"threads_started":1}
I20260619 15:03:29.950655 32473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.949860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.67:60221)
0619 15:03:29.950314 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.950426 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":319,"threads_started":1}
I20260619 15:03:29.953670 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:29.952924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.66:49553)
0619 15:03:29.953379 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:29.953506 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":260,"threads_started":1}
I20260619 15:03:30.944603 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:30.943823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.65:56701)
0619 15:03:30.944257 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:30.944349 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260619 15:03:30.952267 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:30.951466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.67:51749)
0619 15:03:30.951985 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:30.952110 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":312,"threads_started":1}
I20260619 15:03:30.955305 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:30.954516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.66:51467)
0619 15:03:30.955026 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:30.955145 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":263,"threads_started":1}
I20260619 15:03:31.309628 32478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:03:31.309901 32478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } }
I20260619 15:03:31.310532 32478 leader_election.cc:288] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187), 45ddcfea2edc41979fcb0a55eb3662d7 (127.24.103.124:39403)
I20260619 15:03:31.311738 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.310749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60644)
0619 15:03:31.311309 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.311508 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":265,"threads_started":1}
I20260619 15:03:31.311772 31935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" is_pre_election: true
I20260619 15:03:31.312031 31935 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fe5da68b9e34c32aa5758e08a0bdd53 in term 2.
I20260619 15:03:31.312639 31718 leader_election.cc:302] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45ddcfea2edc41979fcb0a55eb3662d7, 8fe5da68b9e34c32aa5758e08a0bdd53; no voters:
I20260619 15:03:31.312916 32478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 15:03:31.313050 32478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:03:31.313161 32478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 2 FOLLOWER]: Advancing to term 3
I20260619 15:03:31.313489 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.313073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60648)
0619 15:03:31.313274 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.313361 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 15:03:31.313664 31718 leader_election.cc:334] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187): Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
I20260619 15:03:31.315480 32478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } }
I20260619 15:03:31.316077 32478 leader_election.cc:288] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [CANDIDATE]: Term 3 election: Requested vote from peers 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187), 45ddcfea2edc41979fcb0a55eb3662d7 (127.24.103.124:39403)
I20260619 15:03:31.316516 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.316155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60650)
0619 15:03:31.316322 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.316388 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 15:03:31.316792 31935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "45ddcfea2edc41979fcb0a55eb3662d7"
I20260619 15:03:31.317022 31935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 2 FOLLOWER]: Advancing to term 3
I20260619 15:03:31.317505 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.317115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60656)
0619 15:03:31.317268 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.317357 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 15:03:31.317670 31718 leader_election.cc:334] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187): Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
I20260619 15:03:31.319336 31935 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fe5da68b9e34c32aa5758e08a0bdd53 in term 3.
I20260619 15:03:31.319758 31718 leader_election.cc:302] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45ddcfea2edc41979fcb0a55eb3662d7, 8fe5da68b9e34c32aa5758e08a0bdd53; no voters: 0e27d27671e74cb798fe7148e4eb0ce8
I20260619 15:03:31.320035 32478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 15:03:31.320443 32478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [term 3 LEADER]: Becoming Leader. State: Replica: 8fe5da68b9e34c32aa5758e08a0bdd53, State: Running, Role: LEADER
I20260619 15:03:31.320817 32478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } }
I20260619 15:03:31.322374 32481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fe5da68b9e34c32aa5758e08a0bdd53. Latest consensus state: current_term: 3 leader_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } } }
I20260619 15:03:31.322669 32481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:31.323359 32483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:31.326769 32483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:31.327674 32483 catalog_manager.cc:1259] Loaded cluster ID: b9aec6e2a07b498ba36c28cb897b5765
I20260619 15:03:31.327744 32483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:31.328919 32483 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:31.329811 32483 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: Loaded TSK: 0
I20260619 15:03:31.330924 32483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:03:31.422406 31730 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:03:31.429651 31730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:03:31.434165 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.433665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60674)
0619 15:03:31.433842 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.433949 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 15:03:31.434533 31935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [term 3 FOLLOWER]: Refusing update from remote peer 8fe5da68b9e34c32aa5758e08a0bdd53: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 15:03:31.435276 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.434890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60680)
0619 15:03:31.435071 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.435168 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 15:03:31.435253 32478 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 15:03:31.435602 31718 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 -> Peer 0e27d27671e74cb798fe7148e4eb0ce8 (127.24.103.126:36187): Couldn't send request to peer 0e27d27671e74cb798fe7148e4eb0ce8. Status: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 15:03:31.437664 32485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fe5da68b9e34c32aa5758e08a0bdd53. Latest consensus state: current_term: 3 leader_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } } }
I20260619 15:03:31.437908 32485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:03:31.439675 32481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } } }
I20260619 15:03:31.440073 32481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:31.440531 32485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0e27d27671e74cb798fe7148e4eb0ce8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36187 } } peers { permanent_uuid: "8fe5da68b9e34c32aa5758e08a0bdd53" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 44261 } attrs { promote: false } } peers { permanent_uuid: "45ddcfea2edc41979fcb0a55eb3662d7" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39403 } attrs { promote: false } } }
I20260619 15:03:31.440805 32485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ddcfea2edc41979fcb0a55eb3662d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:03:31.457683 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.447872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38491 (local address 127.0.0.1:41448)
0619 15:03:31.448056 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.448093 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:31.448200 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:31.448973 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:31.448981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:31.449008 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:31.449413 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:31.449426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.451399 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.451407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:31.454183 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:31.454207 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.456594 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.456603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:31.456886 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.456942 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 15:03:31.457026 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 15:03:31.457227 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 15:03:31.459026 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.448124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.24.103.65:38491)
0619 15:03:31.448609 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:31.448618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:31.448637 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:03:31.448706 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:31.448711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:31.448796 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:31.449558 (+ 762us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:31.449760 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.451210 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.454429 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.456310 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.458149 (+ 1839us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.458317 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:31.458385 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 15:03:31.458594 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":417,"threads_started":1}
I20260619 15:03:31.460325 32039 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3ac098bec94cb1811e07455d381e12 (DEFAULT_TABLE table=test-table [id=7210022fd6e2411fba796d025cf015c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:31.460812 32039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3ac098bec94cb1811e07455d381e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:31.466861 32496 tablet_bootstrap.cc:492] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102: Bootstrap starting.
I20260619 15:03:31.467541 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.449137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.24.103.67:46311)
0619 15:03:31.459699 (+ 10562us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:31.459703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:31.459719 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:03:31.459776 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:31.459779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:31.459846 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:31.459969 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:31.460737 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.461778 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.464414 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.465631 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.465757 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.465869 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:31.467100 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0619 15:03:31.467259 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10452,"thread_start_us":10335,"threads_started":1}
I20260619 15:03:31.467757 32492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.448746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34037 (local address 127.0.0.1:56136)
0619 15:03:31.456394 (+ 7648us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.456415 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:31.456528 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:31.460600 (+ 4072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:31.460605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:31.460631 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:31.460941 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:31.460950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.461882 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.461887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:31.465308 (+ 3421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:31.465322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.466984 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.466990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:31.467147 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.467181 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 15:03:31.467246 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 15:03:31.467372 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7507,"mutex_wait_us":82,"thread_start_us":541,"threads_started":1}
I20260619 15:03:31.467801 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.448701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:46311 (local address 127.0.0.1:34850)
0619 15:03:31.455813 (+ 7112us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.455839 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:31.456062 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:31.460141 (+ 4079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:31.460145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:31.460171 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:31.460500 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:31.460513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.461987 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.461997 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:31.464253 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:31.464266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.465977 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.465984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:31.466182 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.466218 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 15:03:31.466307 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 15:03:31.466452 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6928,"mutex_wait_us":73,"thread_start_us":607,"threads_started":1}
I20260619 15:03:31.468636 32496 tablet_bootstrap.cc:654] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:31.468703 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.449117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.24.103.66:34037)
0619 15:03:31.460264 (+ 11147us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:31.460270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:31.460289 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:03:31.460358 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:31.460365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:31.460462 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:31.460593 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:31.461096 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.461748 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.465478 (+ 3730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.466719 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.468072 (+ 1353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.468234 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:31.468286 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 15:03:31.468416 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11026,"thread_start_us":11590,"threads_started":1}
I20260619 15:03:31.468596 32320 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3ac098bec94cb1811e07455d381e12 (DEFAULT_TABLE table=test-table [id=7210022fd6e2411fba796d025cf015c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:31.469101 32320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3ac098bec94cb1811e07455d381e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:31.469441 32179 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3ac098bec94cb1811e07455d381e12 (DEFAULT_TABLE table=test-table [id=7210022fd6e2411fba796d025cf015c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:31.469882 32179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3ac098bec94cb1811e07455d381e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:31.476543 32498 tablet_bootstrap.cc:492] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5: Bootstrap starting.
I20260619 15:03:31.477437 32499 tablet_bootstrap.cc:492] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9: Bootstrap starting.
I20260619 15:03:31.478302 32496 tablet_bootstrap.cc:492] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102: No bootstrap required, opened a new log
I20260619 15:03:31.478475 32496 ts_tablet_manager.cc:1397] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260619 15:03:31.478530 32498 tablet_bootstrap.cc:654] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:31.479373 32496 raft_consensus.cc:348] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:31.479686 32496 raft_consensus.cc:374] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:31.479795 32496 raft_consensus.cc:729] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c90e3825b54f8f91d82c5ba5e9c102, State: Initialized, Role: FOLLOWER
I20260619 15:03:31.480072 32496 consensus_queue.cc:260] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:31.481384 32496 ts_tablet_manager.cc:1428] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 15:03:31.481469 32499 tablet_bootstrap.cc:654] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:31.481592 32498 tablet_bootstrap.cc:492] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5: No bootstrap required, opened a new log
I20260619 15:03:31.481796 32498 ts_tablet_manager.cc:1397] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 15:03:31.482633 32498 raft_consensus.cc:348] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:31.482945 32498 raft_consensus.cc:374] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:31.483094 32498 raft_consensus.cc:729] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f29062c40dd49a8952f4d893bec9fb5, State: Initialized, Role: FOLLOWER
I20260619 15:03:31.483417 32498 consensus_queue.cc:260] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:31.485910 32498 ts_tablet_manager.cc:1428] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260619 15:03:31.486248 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.481690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.65:41915)
0619 15:03:31.485831 (+ 4141us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.485967 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4008,"thread_start_us":3877,"threads_started":1}
I20260619 15:03:31.486726 32106 heartbeater.cc:503] Master 127.24.103.125:44261 was elected leader, sending a full tablet report...
I20260619 15:03:31.488628 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.487752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.65:58013)
0619 15:03:31.488382 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.488473 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507}
I20260619 15:03:31.489344 32499 tablet_bootstrap.cc:492] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9: No bootstrap required, opened a new log
I20260619 15:03:31.489528 32499 ts_tablet_manager.cc:1397] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260619 15:03:31.490514 32499 raft_consensus.cc:348] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:31.490860 32499 raft_consensus.cc:374] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:31.490995 32499 raft_consensus.cc:729] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638989ea9b4c4df599319b3f97e1b5d9, State: Initialized, Role: FOLLOWER
I20260619 15:03:31.491101 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.487120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.67:60731)
0619 15:03:31.490281 (+ 3161us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.490810 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3031,"thread_start_us":2921,"threads_started":1}
I20260619 15:03:31.491277 32499 consensus_queue.cc:260] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:31.492249 32387 heartbeater.cc:503] Master 127.24.103.125:44261 was elected leader, sending a full tablet report...
I20260619 15:03:31.492575 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.491936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.67:42845)
0619 15:03:31.492358 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.492439 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321}
I20260619 15:03:31.492610 32499 ts_tablet_manager.cc:1428] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 15:03:31.493918 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.492303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.66:47041)
0619 15:03:31.493553 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.493712 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1155,"thread_start_us":1067,"threads_started":1}
I20260619 15:03:31.495492 32246 heartbeater.cc:503] Master 127.24.103.125:44261 was elected leader, sending a full tablet report...
I20260619 15:03:31.495633 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.494532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.66:52479)
0619 15:03:31.495156 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.495325 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":309,"threads_started":1}
W20260619 15:03:31.532848 32247 tablet.cc:2369] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:03:31.537804 32504 raft_consensus.cc:482] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:03:31.538056 32504 raft_consensus.cc:504] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:31.539798 32504 leader_election.cc:288] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c90e3825b54f8f91d82c5ba5e9c102 (127.24.103.65:38491), 638989ea9b4c4df599319b3f97e1b5d9 (127.24.103.66:34037)
I20260619 15:03:31.546005 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.540219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:52939 (local address 127.24.103.65:38491)
0619 15:03:31.540403 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:31.540410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:31.540996 (+ 586us) server_negotiation.cc:407] Connection header received
0619 15:03:31.541166 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:31.541171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:31.541244 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:31.541340 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:31.541887 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.542588 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.544298 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.545071 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.545288 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.545370 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:31.545629 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 15:03:31.545754 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 15:03:31.546003 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.540219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:52077 (local address 127.24.103.66:34037)
0619 15:03:31.540425 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:31.540431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:31.540541 (+ 110us) server_negotiation.cc:407] Connection header received
0619 15:03:31.540878 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:31.540885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:31.540977 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:31.541081 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:31.541589 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.542391 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.544188 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.545017 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.545201 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.545329 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:31.545587 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 15:03:31.545724 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 15:03:31.546038 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.540099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34037 (local address 127.24.103.67:52077)
0619 15:03:31.540396 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.540427 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:31.540694 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:31.541126 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:31.541131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:31.541166 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:31.541429 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:31.541443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.542532 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.542537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:31.544018 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:31.544030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.545211 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.545217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:31.545432 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.545456 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 15:03:31.545533 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 15:03:31.545710 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":4}
I20260619 15:03:31.546267 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.540139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38491 (local address 127.24.103.67:52939)
0619 15:03:31.540875 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.540893 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:31.541024 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:31.541367 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:31.541370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:31.541392 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:31.541737 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:31.541747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.542725 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.542728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:31.544153 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:31.544169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.545309 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.545315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:31.545457 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.545478 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 15:03:31.545601 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 15:03:31.545723 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":18,"thread_start_us":404,"threads_started":1}
I20260619 15:03:31.546708 32199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3ac098bec94cb1811e07455d381e12" candidate_uuid: "8f29062c40dd49a8952f4d893bec9fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638989ea9b4c4df599319b3f97e1b5d9" is_pre_election: true
I20260619 15:03:31.546854 32059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3ac098bec94cb1811e07455d381e12" candidate_uuid: "8f29062c40dd49a8952f4d893bec9fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" is_pre_election: true
I20260619 15:03:31.547107 32199 raft_consensus.cc:2454] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f29062c40dd49a8952f4d893bec9fb5 in term 0.
I20260619 15:03:31.547119 32059 raft_consensus.cc:2454] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f29062c40dd49a8952f4d893bec9fb5 in term 0.
I20260619 15:03:31.547629 32273 leader_election.cc:302] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c90e3825b54f8f91d82c5ba5e9c102, 8f29062c40dd49a8952f4d893bec9fb5; no voters:
I20260619 15:03:31.548025 32504 raft_consensus.cc:2781] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:03:31.548148 32504 raft_consensus.cc:482] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:03:31.548269 32504 raft_consensus.cc:3037] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:31.550396 32504 raft_consensus.cc:504] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:31.551111 32504 leader_election.cc:288] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [CANDIDATE]: Term 1 election: Requested vote from peers 76c90e3825b54f8f91d82c5ba5e9c102 (127.24.103.65:38491), 638989ea9b4c4df599319b3f97e1b5d9 (127.24.103.66:34037)
I20260619 15:03:31.551667 32199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3ac098bec94cb1811e07455d381e12" candidate_uuid: "8f29062c40dd49a8952f4d893bec9fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638989ea9b4c4df599319b3f97e1b5d9"
I20260619 15:03:31.551689 32059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3ac098bec94cb1811e07455d381e12" candidate_uuid: "8f29062c40dd49a8952f4d893bec9fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c90e3825b54f8f91d82c5ba5e9c102"
I20260619 15:03:31.551896 32059 raft_consensus.cc:3037] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:31.551916 32199 raft_consensus.cc:3037] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:31.554976 32059 raft_consensus.cc:2454] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f29062c40dd49a8952f4d893bec9fb5 in term 1.
I20260619 15:03:31.554968 32199 raft_consensus.cc:2454] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f29062c40dd49a8952f4d893bec9fb5 in term 1.
I20260619 15:03:31.555552 32276 leader_election.cc:302] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 638989ea9b4c4df599319b3f97e1b5d9, 76c90e3825b54f8f91d82c5ba5e9c102, 8f29062c40dd49a8952f4d893bec9fb5; no voters:
I20260619 15:03:31.555943 32504 raft_consensus.cc:2781] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:31.556480 32504 raft_consensus.cc:686] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 1 LEADER]: Becoming Leader. State: Replica: 8f29062c40dd49a8952f4d893bec9fb5, State: Running, Role: LEADER
I20260619 15:03:31.557011 32504 consensus_queue.cc:237] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } } peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } }
I20260619 15:03:31.557524 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.557047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.67:41567)
0619 15:03:31.557218 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.557348 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 15:03:31.560847 31731 catalog_manager.cc:5495] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f29062c40dd49a8952f4d893bec9fb5 (127.24.103.67). New cstate: current_term: 1 leader_uuid: "8f29062c40dd49a8952f4d893bec9fb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f29062c40dd49a8952f4d893bec9fb5" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46311 } health_report { overall_health: HEALTHY } } }
I20260619 15:03:31.594020 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.589242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.24.103.67:46311)
0619 15:03:31.589449 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:31.589453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:31.589471 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:03:31.589660 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:31.589664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:31.589747 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:31.589884 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:03:31.590195 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.590973 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.592199 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.592570 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.592668 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.592998 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:03:31.593528 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:03:31.593654 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:31.593751 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 15:03:31.593819 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:03:31.598260 32059 raft_consensus.cc:1261] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102 [term 1 FOLLOWER]: Refusing update from remote peer 8f29062c40dd49a8952f4d893bec9fb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:31.598548 32199 raft_consensus.cc:1261] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9 [term 1 FOLLOWER]: Refusing update from remote peer 8f29062c40dd49a8952f4d893bec9fb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:31.599152 32510 consensus_queue.cc:1035] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c90e3825b54f8f91d82c5ba5e9c102" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:31.599550 32504 consensus_queue.cc:1035] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "638989ea9b4c4df599319b3f97e1b5d9" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:31.602208 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.601497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.65:40179)
0619 15:03:31.601816 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.601960 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260619 15:03:31.603125 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.602549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.66:45561)
0619 15:03:31.602855 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.602959 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 15:03:31.612470 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.611864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.66:36169)
0619 15:03:31.612121 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.612265 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260619 15:03:31.615808 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.615298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.65:46395)
0619 15:03:31.615543 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.615638 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260619 15:03:31.622887 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.622136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.67:49281)
0619 15:03:31.622551 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.622674 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
I20260619 15:03:31.624475 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.623591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.24.103.67:45535)
0619 15:03:31.624225 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:31.624338 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539}
I20260619 15:03:31.640044 31730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:31.640399 31730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:31.646584 31730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fe5da68b9e34c32aa5758e08a0bdd53: Sending DeleteTablet for 3 replicas of tablet 9b3ac098bec94cb1811e07455d381e12
I20260619 15:03:31.647572 32039 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b3ac098bec94cb1811e07455d381e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:31 UTC) from {username='oryx', principal='oryx/127.24.103.125@KRBTEST.COM'} at 127.0.0.1:41448
I20260619 15:03:31.647836 32320 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b3ac098bec94cb1811e07455d381e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:31 UTC) from {username='oryx', principal='oryx/127.24.103.125@KRBTEST.COM'} at 127.0.0.1:34850
I20260619 15:03:31.648882 32179 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b3ac098bec94cb1811e07455d381e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:31 UTC) from {username='oryx', principal='oryx/127.24.103.125@KRBTEST.COM'} at 127.0.0.1:56136
I20260619 15:03:31.651618 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 31914
I20260619 15:03:31.667858 32525 tablet_replica.cc:330] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9: stopping tablet replica
I20260619 15:03:31.668293 32525 raft_consensus.cc:2229] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:31.668792 32525 raft_consensus.cc:2258] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:31.672063 32523 tablet_replica.cc:330] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102: stopping tablet replica
I20260619 15:03:31.672441 32523 raft_consensus.cc:2229] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:31.673434 32525 ts_tablet_manager.cc:1905] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:31.673906 32523 raft_consensus.cc:2258] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:31.676209 32523 ts_tablet_manager.cc:1905] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:31.679106 32524 tablet_replica.cc:330] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5: stopping tablet replica
I20260619 15:03:31.679323 32524 raft_consensus.cc:2229] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:03:31.680004 32524 raft_consensus.cc:2258] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:31.682106 32524 ts_tablet_manager.cc:1905] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:31.685853 32525 ts_tablet_manager.cc:1918] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:31.686021 32525 log.cc:1196] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/9b3ac098bec94cb1811e07455d381e12
I20260619 15:03:31.686491 32525 ts_tablet_manager.cc:1939] T 9b3ac098bec94cb1811e07455d381e12 P 638989ea9b4c4df599319b3f97e1b5d9: Deleting consensus metadata
I20260619 15:03:31.687958 31718 catalog_manager.cc:4841] TS 638989ea9b4c4df599319b3f97e1b5d9 (127.24.103.66:34037): tablet 9b3ac098bec94cb1811e07455d381e12 (table test-table [id=7210022fd6e2411fba796d025cf015c3]) successfully deleted
I20260619 15:03:31.689353 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.653088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.24.103.65:38491)
0619 15:03:31.653280 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:31.653286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:31.664461 (+ 11175us) server_negotiation.cc:407] Connection header received
0619 15:03:31.664714 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:31.664720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:31.664816 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:31.664945 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:31.667308 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.668466 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.671258 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.672049 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.672198 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.680415 (+ 8217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:31.680438 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:31.680460 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:31.680518 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:31.684721 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:31.685740 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:31.685751 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:31.685754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:31.685937 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:31.686433 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:31.686439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:31.686443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:31.686981 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:31.687272 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:31.688676 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0619 15:03:31.688945 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:03:31.689697 32523 ts_tablet_manager.cc:1918] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:31.689895 32523 log.cc:1196] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/9b3ac098bec94cb1811e07455d381e12
I20260619 15:03:31.690375 32523 ts_tablet_manager.cc:1939] T 9b3ac098bec94cb1811e07455d381e12 P 76c90e3825b54f8f91d82c5ba5e9c102: Deleting consensus metadata
I20260619 15:03:31.691120 32099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:31.691272 32524 ts_tablet_manager.cc:1918] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:31.691468 32524 log.cc:1196] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/9b3ac098bec94cb1811e07455d381e12
I20260619 15:03:31.691689 31715 catalog_manager.cc:4841] TS 76c90e3825b54f8f91d82c5ba5e9c102 (127.24.103.65:38491): tablet 9b3ac098bec94cb1811e07455d381e12 (table test-table [id=7210022fd6e2411fba796d025cf015c3]) successfully deleted
I20260619 15:03:31.692155 32524 ts_tablet_manager.cc:1939] T 9b3ac098bec94cb1811e07455d381e12 P 8f29062c40dd49a8952f4d893bec9fb5: Deleting consensus metadata
I20260619 15:03:31.693388 31717 catalog_manager.cc:4841] TS 8f29062c40dd49a8952f4d893bec9fb5 (127.24.103.67:46311): tablet 9b3ac098bec94cb1811e07455d381e12 (table test-table [id=7210022fd6e2411fba796d025cf015c3]) successfully deleted
I20260619 15:03:31.789119 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 31914
I20260619 15:03:31.807649 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 32116
I20260619 15:03:31.823779 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.808601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.24.103.66:34037)
0619 15:03:31.808786 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:31.808792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:31.808813 (+ 21us) server_negotiation.cc:407] Connection header received
0619 15:03:31.808977 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:31.808983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:31.809069 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:31.809188 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:31.810452 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.811681 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.812981 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.813610 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.813740 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.816429 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:31.816449 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:31.816470 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:31.816515 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:31.820586 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:31.821480 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:31.821491 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:31.821494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:31.821650 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:31.822057 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:31.822062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:31.822065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:31.822536 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:31.822881 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:31.823158 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 15:03:31.823379 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:03:31.824678 32239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:31.945946 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 32116
I20260619 15:03:31.968324 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 32257
I20260619 15:03:31.985212 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:31.969211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34866 (local address 127.24.103.67:46311)
0619 15:03:31.969407 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:31.969412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:31.969472 (+ 60us) server_negotiation.cc:407] Connection header received
0619 15:03:31.969751 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:31.969759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:31.969830 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:31.969912 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:31.971885 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.972687 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.974556 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:31.975178 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:31.975276 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:31.978771 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:31.978840 (+ 69us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:31.978860 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:31.978907 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:31.982064 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:31.983125 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:31.983132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:31.983134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:31.983269 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:31.983693 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:31.983696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:31.983698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:31.984076 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:31.984303 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:31.984716 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 15:03:31.984901 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 15:03:31.985960 32380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:32.031790 32530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:32.031060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60686)
0619 15:03:32.031543 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:32.031628 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":270,"threads_started":1}
I20260619 15:03:32.033078 32530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:32.032738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36187 (local address 127.0.0.1:60692)
0619 15:03:32.032880 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:32.032961 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 15:03:32.115049 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 32257
I20260619 15:03:32.135493 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 31588
I20260619 15:03:32.136576 31761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:32.258227 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 31588
W20260619 15:03:32.279368 31908 connection.cc:535] server connection from 127.0.0.1:35860 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 15:03:32.279742 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 31688
I20260619 15:03:32.280813 31955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:32.395709 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 31688
2026-06-19T15:03:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11501 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:03:32 dist-test-slave-zpbw krb5kdc[32536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:03:32 dist-test-slave-zpbw krb5kdc[32536](info): set up 2 sockets
Jun 19 15:03:32 dist-test-slave-zpbw krb5kdc[32536](info): commencing operation
krb5kdc: starting...
W20260619 15:03:33.500669 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:03:33 dist-test-slave-zpbw krb5kdc[32536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:03:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:03:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.126.keytab.
Entry for principal oryx/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.126.keytab.
I20260619 15:03:33.724474 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41781
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.126.keytab
--principal=oryx/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:41781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:34.069146 32549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:34.069461 32549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:34.069552 32549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:34.077014 32549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:34.077139 32549 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 15:03:34.077181 32549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:34.077246 32549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:34.077297 32549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:34.077378 32549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:03:34.087534 32549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:41781
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.126.keytab
--principal=oryx/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:34.088789 32549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:34.090087 32549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:34.101930 32554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:34.101994 32555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:34.102018 32557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:34.102942 32549 server_base.cc:1034] running on GCE node
I20260619 15:03:34.103715 32549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:34.105192 32549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:34.106432 32549 hybrid_clock.cc:648] HybridClock initialized: now 1781881414106354 us; error 58 us; skew 500 ppm
I20260619 15:03:34.106873 32549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:34 dist-test-slave-zpbw krb5kdc[32536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881414, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:34.110011 32549 init.cc:377] Logged in from keytab as oryx/127.24.103.126@KRBTEST.COM (short username oryx)
I20260619 15:03:34.111518 32549 webserver.cc:460] Webserver started at http://127.24.103.126:39869/ using document root <none> and password file <none>
I20260619 15:03:34.112251 32549 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:34.112368 32549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:34.112690 32549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:34.115939 32549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "ea9c434a3d6b4dcd84b0852cf7f7dc81"
format_stamp: "Formatted at 2026-06-19 15:03:34 on dist-test-slave-zpbw"
server_key: "76affb51ec92df8bafec8158c038ebde"
server_key_iv: "4fc9f0c2c92e49544ab3ae246cc59475"
server_key_version: "encryptionkey@0"
I20260619 15:03:34.116772 32549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "ea9c434a3d6b4dcd84b0852cf7f7dc81"
format_stamp: "Formatted at 2026-06-19 15:03:34 on dist-test-slave-zpbw"
server_key: "76affb51ec92df8bafec8158c038ebde"
server_key_iv: "4fc9f0c2c92e49544ab3ae246cc59475"
server_key_version: "encryptionkey@0"
I20260619 15:03:34.122290 32549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 15:03:34.125957 32564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:34.127444 32549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 15:03:34.127696 32549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "ea9c434a3d6b4dcd84b0852cf7f7dc81"
format_stamp: "Formatted at 2026-06-19 15:03:34 on dist-test-slave-zpbw"
server_key: "76affb51ec92df8bafec8158c038ebde"
server_key_iv: "4fc9f0c2c92e49544ab3ae246cc59475"
server_key_version: "encryptionkey@0"
I20260619 15:03:34.127893 32549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:34.149966 32549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:34.154033 32549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:34.154265 32549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:34.176268 32549 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:41781
I20260619 15:03:34.176275 32625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:41781 every 8 connection(s)
I20260619 15:03:34.177724 32549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:34.181607 32626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:34.186511 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 32549
I20260619 15:03:34.186748 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:03:34.187409 24989 external_mini_cluster.cc:1428] Setting key 5c85d17bc6b8f5a185c6ab72ea12c1f4
I20260619 15:03:34.192948 32626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81: Bootstrap starting.
I20260619 15:03:34.196925 32626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:34.198221 32626 log.cc:824] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:34.201843 32626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81: No bootstrap required, opened a new log
Jun 19 15:03:34 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:34.210040 32626 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9c434a3d6b4dcd84b0852cf7f7dc81" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41781 } }
I20260619 15:03:34.210441 32626 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:34.210578 32626 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9c434a3d6b4dcd84b0852cf7f7dc81, State: Initialized, Role: FOLLOWER
I20260619 15:03:34.211264 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:34.189514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.24.103.126:41781)
0619 15:03:34.190363 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:34.190375 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:34.190417 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:03:34.192127 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:34.192150 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:34.192539 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:34.192867 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:34.194739 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:34.196132 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:34.197975 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:34.198900 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:34.199105 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:34.203764 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:34.203773 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:34.203791 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:34.203842 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:34.206975 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:34.207681 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:34.207688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:34.207694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:34.207857 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:34.208243 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:34.208257 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:34.208260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:34.208804 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:34.209143 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:34.209600 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 15:03:34.210033 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":371,"threads_started":1}
I20260619 15:03:34.211345 32626 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9c434a3d6b4dcd84b0852cf7f7dc81" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41781 } }
I20260619 15:03:34.211652 32626 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:34.211795 32626 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:34.212014 32626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:34.215235 32626 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9c434a3d6b4dcd84b0852cf7f7dc81" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41781 } }
I20260619 15:03:34.215749 32626 leader_election.cc:302] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9c434a3d6b4dcd84b0852cf7f7dc81; no voters:
I20260619 15:03:34.216324 32626 leader_election.cc:288] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:03:34.216454 32631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:34.217196 32631 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [term 1 LEADER]: Becoming Leader. State: Replica: ea9c434a3d6b4dcd84b0852cf7f7dc81, State: Running, Role: LEADER
I20260619 15:03:34.217803 32631 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9c434a3d6b4dcd84b0852cf7f7dc81" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41781 } }
I20260619 15:03:34.218554 32626 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:34.220418 32633 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9c434a3d6b4dcd84b0852cf7f7dc81. Latest consensus state: current_term: 1 leader_uuid: "ea9c434a3d6b4dcd84b0852cf7f7dc81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9c434a3d6b4dcd84b0852cf7f7dc81" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41781 } } }
I20260619 15:03:34.220789 32633 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:34.221886 32632 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9c434a3d6b4dcd84b0852cf7f7dc81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9c434a3d6b4dcd84b0852cf7f7dc81" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41781 } } }
I20260619 15:03:34.222219 32632 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81 [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:34.223024 32640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:34.228557 32640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:34.244002 32640 catalog_manager.cc:1347] Generated new cluster ID: 5b207f9aec134b628026626415042721
I20260619 15:03:34.244158 32640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:34.260627 32640 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:03:34.262136 32640 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:34.270476 32640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81: Generated new TSK 0
I20260619 15:03:34.271657 32640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab.
Entry for principal oryx/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab.
I20260619 15:03:34.334444 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41781
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:34.677080 32652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:34.677542 32652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:34.677747 32652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:34.685506 32652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:34.685708 32652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:34.685886 32652 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 15:03:34.686038 32652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:03:34.696516 32652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:41781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:34.698096 32652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:34.699617 32652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:34.713955 32660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:34.713989 32658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:34.713899 32657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:34.714888 32652 server_base.cc:1034] running on GCE node
I20260619 15:03:34.715605 32652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:34.716519 32652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:34.717767 32652 hybrid_clock.cc:648] HybridClock initialized: now 1781881414717693 us; error 49 us; skew 500 ppm
I20260619 15:03:34.718055 32652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:34 dist-test-slave-zpbw krb5kdc[32536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881414, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:34.721822 32652 init.cc:377] Logged in from keytab as oryx/127.24.103.65@KRBTEST.COM (short username oryx)
I20260619 15:03:34.723570 32652 webserver.cc:460] Webserver started at http://127.24.103.65:45313/ using document root <none> and password file <none>
I20260619 15:03:34.724346 32652 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:34.724439 32652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:34.724815 32652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:34.728614 32652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "fdf49c9af084419b9f46614f84987906"
format_stamp: "Formatted at 2026-06-19 15:03:34 on dist-test-slave-zpbw"
server_key: "4d871e439deedab35c3f0475414bcd24"
server_key_iv: "461e13ae396a795d6bb0d4101d8d17d8"
server_key_version: "encryptionkey@0"
I20260619 15:03:34.729580 32652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "fdf49c9af084419b9f46614f84987906"
format_stamp: "Formatted at 2026-06-19 15:03:34 on dist-test-slave-zpbw"
server_key: "4d871e439deedab35c3f0475414bcd24"
server_key_iv: "461e13ae396a795d6bb0d4101d8d17d8"
server_key_version: "encryptionkey@0"
I20260619 15:03:34.735277 32652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 15:03:34.738968 32667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:34.740327 32652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:03:34.740526 32652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "fdf49c9af084419b9f46614f84987906"
format_stamp: "Formatted at 2026-06-19 15:03:34 on dist-test-slave-zpbw"
server_key: "4d871e439deedab35c3f0475414bcd24"
server_key_iv: "461e13ae396a795d6bb0d4101d8d17d8"
server_key_version: "encryptionkey@0"
I20260619 15:03:34.740767 32652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:34.756745 32652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:34.761593 32652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:34.761900 32652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:34.764721 32652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:34.764858 32652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:34.765031 32652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:34.765113 32652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 15:03:34 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881414, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:34.790678 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:34.765509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:34003 (local address 127.24.103.126:41781)
0619 15:03:34.766115 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:34.766120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:34.767445 (+ 1325us) server_negotiation.cc:407] Connection header received
0619 15:03:34.769494 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:34.769499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:34.769611 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:34.769758 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:34.772648 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:34.773616 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:34.776329 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:34.777062 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:34.777232 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:34.782121 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:34.782130 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:34.782138 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:34.782190 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:34.786092 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:34.787243 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:34.787249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:34.787253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:34.787391 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:34.787990 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:34.787996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:34.787999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:34.788391 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:34.788538 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:34.789722 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0619 15:03:34.790069 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":376,"threads_started":1}
I20260619 15:03:34.791308 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:34.766294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41781 (local address 127.24.103.65:34003)
0619 15:03:34.767264 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:34.767310 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:34.769196 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:34.770148 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:34.770166 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:34.771175 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:34.772391 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:34.772428 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:34.773797 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:34.773801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:34.776104 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:34.776127 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:34.777330 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:34.777340 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:34.777673 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:34.778574 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:34.778610 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:34.781797 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:34.786303 (+ 4506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:34.786314 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:34.786332 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:34.787095 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:34.787509 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:34.787514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:34.787520 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:34.787869 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:34.788620 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:34.788627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:34.789195 (+ 568us) client_negotiation.cc:769] Sending connection context
0619 15:03:34.789539 (+ 344us) client_negotiation.cc:240] Negotiation successful
0619 15:03:34.790029 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":493,"threads_started":1}
I20260619 15:03:34.817658 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:34.797701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:49639 (local address 127.24.103.126:41781)
0619 15:03:34.797928 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:34.797938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:34.798533 (+ 595us) server_negotiation.cc:407] Connection header received
0619 15:03:34.798885 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:34.798894 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:34.799027 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:34.799118 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:34.801052 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:34.802459 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:34.805191 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:34.806140 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:34.806261 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:34.809548 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:34.809560 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:34.809568 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:34.809623 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:34.813878 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:34.815268 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:34.815276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:34.815282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:34.815477 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:34.816020 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:34.816028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:34.816033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:34.816384 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:34.816577 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:34.817014 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 15:03:34.817266 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 15:03:34.817765 32741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:34.797667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41781 (local address 127.24.103.65:49639)
0619 15:03:34.798408 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:34.798435 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:34.798677 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:34.799507 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:34.799514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:34.800302 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:34.800777 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:34.800793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:34.802657 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:34.802663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:34.804936 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:34.804960 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:34.806624 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:34.806633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:34.806943 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:34.807779 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:34.807812 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:34.809214 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:34.814117 (+ 4903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:34.814126 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:34.814130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:34.815065 (+ 935us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:34.815653 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:34.815661 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:34.815667 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:34.815857 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:34.816528 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:34.816534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:34.816746 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 15:03:34.816902 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 15:03:34.817264 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":1497,"threads_started":1}
I20260619 15:03:34.830371 32652 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:38495
I20260619 15:03:34.830426 319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:38495 every 8 connection(s)
I20260619 15:03:34.832257 32652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:03:34.834388 320 heartbeater.cc:347] Connected to a master server at 127.24.103.126:41781
I20260619 15:03:34.834601 320 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:34.835327 320 heartbeater.cc:511] Master 127.24.103.126:41781 requested a full tablet report, sending...
I20260619 15:03:34.837301 32581 ts_manager.cc:194] Registered new tserver with Master: fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495)
I20260619 15:03:34.838923 32581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:34003
I20260619 15:03:34.842464 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 32652
I20260619 15:03:34.842634 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:03:34.843283 24989 external_mini_cluster.cc:1428] Setting key 67ad3469b7c4f09976152e5f6b61e70e
WARNING: no policy specified for oryx/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab.
Entry for principal oryx/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab.
I20260619 15:03:34.892099 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41781
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:35.224568 324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:35.224859 324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:35.224956 324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:35.232621 324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:35.232786 324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:35.232870 324 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 15:03:35.232951 324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:03:35.243440 324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:41781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:35.244808 324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:35.246197 324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:35.260105 332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:35.260038 330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:35.260136 329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:35.261044 324 server_base.cc:1034] running on GCE node
I20260619 15:03:35.261816 324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:35.262938 324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:35.264186 324 hybrid_clock.cc:648] HybridClock initialized: now 1781881415264122 us; error 43 us; skew 500 ppm
I20260619 15:03:35.264508 324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:35 dist-test-slave-zpbw krb5kdc[32536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881415, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:35.267678 324 init.cc:377] Logged in from keytab as oryx/127.24.103.66@KRBTEST.COM (short username oryx)
I20260619 15:03:35.269194 324 webserver.cc:460] Webserver started at http://127.24.103.66:43677/ using document root <none> and password file <none>
I20260619 15:03:35.269928 324 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:35.270064 324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:35.270381 324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:35.273430 324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "7fbc87acce8748a49111dd15c74e5f6a"
format_stamp: "Formatted at 2026-06-19 15:03:35 on dist-test-slave-zpbw"
server_key: "39b2eb3b0dde179c19a6090e11155683"
server_key_iv: "1be06bf1a95afc629bab84e93ac185b1"
server_key_version: "encryptionkey@0"
I20260619 15:03:35.274256 324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "7fbc87acce8748a49111dd15c74e5f6a"
format_stamp: "Formatted at 2026-06-19 15:03:35 on dist-test-slave-zpbw"
server_key: "39b2eb3b0dde179c19a6090e11155683"
server_key_iv: "1be06bf1a95afc629bab84e93ac185b1"
server_key_version: "encryptionkey@0"
I20260619 15:03:35.279517 324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 15:03:35.283345 339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:35.284799 324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 15:03:35.284987 324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "7fbc87acce8748a49111dd15c74e5f6a"
format_stamp: "Formatted at 2026-06-19 15:03:35 on dist-test-slave-zpbw"
server_key: "39b2eb3b0dde179c19a6090e11155683"
server_key_iv: "1be06bf1a95afc629bab84e93ac185b1"
server_key_version: "encryptionkey@0"
I20260619 15:03:35.285229 324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:35.301287 324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:35.305675 324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:35.305956 324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:35.308806 324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:35.308920 324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:35.309106 324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:35.309228 324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 15:03:35 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881415, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:35.335031 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.310104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41781 (local address 127.24.103.66:38027)
0619 15:03:35.311470 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:35.311525 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:35.313545 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:35.314355 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:35.314370 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:35.315292 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:35.316275 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:35.316308 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.317840 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.317849 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:35.320712 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:35.320731 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.321656 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.321666 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:35.322091 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:35.323125 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:35.323152 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:35.326065 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:35.329793 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:35.329803 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:35.329825 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:35.330513 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:35.331062 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:35.331076 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:35.331081 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:35.331356 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:35.332198 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:35.332213 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:35.332631 (+ 418us) client_negotiation.cc:769] Sending connection context
0619 15:03:35.333028 (+ 397us) client_negotiation.cc:240] Negotiation successful
0619 15:03:35.333662 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1085,"thread_start_us":804,"threads_started":1}
I20260619 15:03:35.335647 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.309370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:38027 (local address 127.24.103.126:41781)
0619 15:03:35.309574 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:35.309581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:35.311682 (+ 2101us) server_negotiation.cc:407] Connection header received
0619 15:03:35.313851 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:35.313856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:35.313950 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:35.314074 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:35.316478 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.317637 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.320891 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.321372 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.321494 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:35.326308 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:35.326313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:35.326316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:35.326350 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:35.329584 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:35.330734 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:35.330743 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:35.330749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:35.330963 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:35.331538 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:35.331545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:35.331551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:35.331878 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:35.332099 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:35.335019 (+ 2920us) server_negotiation.cc:299] Negotiation successful
0619 15:03:35.335311 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 15:03:35.355079 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.341418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:47811 (local address 127.24.103.126:41781)
0619 15:03:35.341684 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:35.341693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:35.342209 (+ 516us) server_negotiation.cc:407] Connection header received
0619 15:03:35.342369 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:35.342377 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:35.342483 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:35.342645 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:35.343725 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.344816 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.346445 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.346933 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.347089 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:35.349540 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:35.349544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:35.349548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:35.349581 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:35.352303 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:35.353004 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:35.353008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:35.353011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:35.353153 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:35.353518 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:35.353524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:35.353527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:35.353861 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:35.354023 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:35.354433 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 15:03:35.354677 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:03:35.355266 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.341443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41781 (local address 127.24.103.66:47811)
0619 15:03:35.342101 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:35.342121 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:35.342245 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:35.342660 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:35.342663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:35.343191 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:35.343541 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:35.343550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.344938 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.344945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:35.346298 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:35.346310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.347138 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.347148 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:35.347328 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:35.348075 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:35.348095 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:35.349329 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:35.352450 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:35.352456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:35.352460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:35.352857 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:35.353270 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:35.353275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:35.353279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:35.353408 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:35.353984 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:35.353989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:35.354181 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 15:03:35.354336 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 15:03:35.354718 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":364,"threads_started":1}
I20260619 15:03:35.366220 324 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:44825
I20260619 15:03:35.366240 458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:44825 every 8 connection(s)
I20260619 15:03:35.367583 324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:03:35.369807 459 heartbeater.cc:347] Connected to a master server at 127.24.103.126:41781
I20260619 15:03:35.370036 459 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:35.370689 459 heartbeater.cc:511] Master 127.24.103.126:41781 requested a full tablet report, sending...
I20260619 15:03:35.372012 32581 ts_manager.cc:194] Registered new tserver with Master: 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825)
I20260619 15:03:35.372824 32581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:38027
I20260619 15:03:35.374891 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 324
I20260619 15:03:35.375064 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:03:35.375538 24989 external_mini_cluster.cc:1428] Setting key 1398c11127f43db6338c23243b3f7ca9
WARNING: no policy specified for oryx/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab.
Entry for principal oryx/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab.
I20260619 15:03:35.422118 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41781
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:35.756613 463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:35.756990 463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:35.757174 463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:35.764652 463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:35.764858 463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:35.765043 463 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 15:03:35.765205 463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:03:35.775501 463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:41781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:35.776911 463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:35.778388 463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:35.792654 468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:35.792763 471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:35.792709 469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:35.793741 463 server_base.cc:1034] running on GCE node
I20260619 15:03:35.794461 463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:35.795508 463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:35.796732 463 hybrid_clock.cc:648] HybridClock initialized: now 1781881415796651 us; error 56 us; skew 500 ppm
I20260619 15:03:35.797043 463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:35 dist-test-slave-zpbw krb5kdc[32536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881415, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:35.800202 463 init.cc:377] Logged in from keytab as oryx/127.24.103.67@KRBTEST.COM (short username oryx)
I20260619 15:03:35.801680 463 webserver.cc:460] Webserver started at http://127.24.103.67:39003/ using document root <none> and password file <none>
I20260619 15:03:35.802423 463 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:35.802556 463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:35.802955 463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:35.805898 463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "9a821c4a3b474f25b5241f603dfb694a"
format_stamp: "Formatted at 2026-06-19 15:03:35 on dist-test-slave-zpbw"
server_key: "4e97e754f0a13f5b770c5ca5c231e009"
server_key_iv: "68d794f3cde6041ba5e968c2c52f13cd"
server_key_version: "encryptionkey@0"
I20260619 15:03:35.806722 463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "9a821c4a3b474f25b5241f603dfb694a"
format_stamp: "Formatted at 2026-06-19 15:03:35 on dist-test-slave-zpbw"
server_key: "4e97e754f0a13f5b770c5ca5c231e009"
server_key_iv: "68d794f3cde6041ba5e968c2c52f13cd"
server_key_version: "encryptionkey@0"
I20260619 15:03:35.812079 463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 15:03:35.815977 478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:35.817806 463 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 15:03:35.818006 463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "9a821c4a3b474f25b5241f603dfb694a"
format_stamp: "Formatted at 2026-06-19 15:03:35 on dist-test-slave-zpbw"
server_key: "4e97e754f0a13f5b770c5ca5c231e009"
server_key_iv: "68d794f3cde6041ba5e968c2c52f13cd"
server_key_version: "encryptionkey@0"
I20260619 15:03:35.818306 463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:35.841782 320 heartbeater.cc:503] Master 127.24.103.126:41781 was elected leader, sending a full tablet report...
I20260619 15:03:35.842240 463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:35.846872 463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:35.847105 463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:35.849845 463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:35.849942 463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:35.850069 463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:35.850147 463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 15:03:35 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881415, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:35.875651 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.851669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:52181 (local address 127.24.103.126:41781)
0619 15:03:35.851888 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:35.851897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:35.853332 (+ 1435us) server_negotiation.cc:407] Connection header received
0619 15:03:35.856796 (+ 3464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:35.856804 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:35.856918 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:35.857027 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:35.858734 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.859950 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.861957 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.862593 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.862746 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:35.867736 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:35.867745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:35.867752 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:35.867803 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:35.871227 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:35.871987 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:35.871995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:35.871999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:35.872175 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:35.872752 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:35.872758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:35.872763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:35.873202 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:35.873363 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:35.874840 (+ 1477us) server_negotiation.cc:299] Negotiation successful
0619 15:03:35.875180 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:03:35.877625 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.852212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41781 (local address 127.24.103.67:52181)
0619 15:03:35.853135 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:35.853197 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:35.856393 (+ 3196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:35.857298 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:35.857306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:35.857882 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:35.858532 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:35.858567 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.860123 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.860129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:35.861810 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:35.861823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.862903 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.862916 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:35.863327 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:35.864190 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:35.864231 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:35.867476 (+ 3245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:35.871375 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:35.871389 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:35.871407 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:35.871858 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:35.872313 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:35.872320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:35.872327 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:35.872616 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:35.873373 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:35.873386 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:35.874223 (+ 837us) client_negotiation.cc:769] Sending connection context
0619 15:03:35.874678 (+ 455us) client_negotiation.cc:240] Negotiation successful
0619 15:03:35.875519 (+ 841us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":349,"threads_started":1}
I20260619 15:03:35.898528 463 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:37143
I20260619 15:03:35.898458 597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:37143 every 8 connection(s)
I20260619 15:03:35.900269 463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:03:35.901340 571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.885531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41781 (local address 127.24.103.67:34371)
0619 15:03:35.886559 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:35.886584 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:35.886760 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:35.887228 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:35.887238 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:35.887779 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:35.888086 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:35.888097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.889705 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.889713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:35.891468 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:35.891479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.892213 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.892219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:35.892405 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:35.893226 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:35.893244 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:35.894011 (+ 767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:35.898553 (+ 4542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:35.898559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:35.898565 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:35.899097 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:35.899520 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:35.899524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:35.899526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:35.899639 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:35.900271 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:35.900274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:35.900427 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 15:03:35.900533 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 15:03:35.900771 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":655,"threads_started":1}
I20260619 15:03:35.901469 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.885865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:34371 (local address 127.24.103.126:41781)
0619 15:03:35.886124 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:35.886133 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:35.886700 (+ 567us) server_negotiation.cc:407] Connection header received
0619 15:03:35.886938 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:35.886946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:35.887041 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:35.887173 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:35.888243 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.889535 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.891623 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.892024 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.892172 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:35.894272 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:35.894282 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:35.894290 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:35.894344 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:35.898393 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:35.899226 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:35.899232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:35.899235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:35.899408 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:35.899763 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:35.899769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:35.899773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:35.900152 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:35.900315 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:35.900680 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 15:03:35.900974 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 15:03:35.902626 598 heartbeater.cc:347] Connected to a master server at 127.24.103.126:41781
I20260619 15:03:35.903105 598 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:35.903769 598 heartbeater.cc:511] Master 127.24.103.126:41781 requested a full tablet report, sending...
I20260619 15:03:35.904881 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 463
I20260619 15:03:35.905092 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:03:35.905478 32581 ts_manager.cc:194] Registered new tserver with Master: 9a821c4a3b474f25b5241f603dfb694a (127.24.103.67:37143)
I20260619 15:03:35.905542 24989 external_mini_cluster.cc:1428] Setting key 64bdcd7eda8b15715d26768fe81bca23
I20260619 15:03:35.906380 32581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:52181
I20260619 15:03:35.908450 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 15:03:35 dist-test-slave-zpbw krb5kdc[32536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881415, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 15:03:35 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:35 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 15:03:35.958362 32676 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:35.948005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.24.103.126:41781)
0619 15:03:35.948236 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:35.948255 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:35.948437 (+ 182us) server_negotiation.cc:407] Connection header received
0619 15:03:35.948596 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:35.948600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:35.948666 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:35.948753 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:35.950092 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.950929 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.952162 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.952584 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.952681 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:35.958128 (+ 5447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57688: BlockingRecv error: recv got EOF from 127.0.0.1:57688 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 15:03:35 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881415, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:35.974341 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.961558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.24.103.126:41781)
0619 15:03:35.961713 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:35.961717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:35.962040 (+ 323us) server_negotiation.cc:407] Connection header received
0619 15:03:35.962171 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:35.962175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:35.962249 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:35.962333 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:35.963546 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.964253 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.965431 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:35.965882 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:35.965970 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:35.969533 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:35.969538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:35.969541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:35.969580 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:35.972253 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:35.972932 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:35.972935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:35.972937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:35.973046 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:35.973341 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:35.973344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:35.973346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:35.973592 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:35.973800 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:35.973961 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 15:03:35.974128 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 15:03:35.978590 32581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:03:35.985384 32581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:03:36.022239 617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.998056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.24.103.65:38495)
0619 15:03:36.010484 (+ 12428us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.010500 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.011861 (+ 1361us) server_negotiation.cc:407] Connection header received
0619 15:03:36.012004 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.012016 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.012353 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.012602 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:36.013687 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.015774 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.019324 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.020872 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.021003 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.021221 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.021783 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.021962 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12174,"thread_start_us":13552,"threads_started":1}
I20260619 15:03:36.022279 618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.997858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38495 (local address 127.0.0.1:35866)
0619 15:03:36.011152 (+ 13294us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:36.011205 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:36.011411 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:36.012918 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:36.012931 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:36.012976 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:36.013432 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.013448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.015947 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.015956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.018459 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.018472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.021097 (+ 2625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.021105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.021318 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.021370 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 15:03:36.021503 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 15:03:36.021815 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13043,"mutex_wait_us":66,"thread_start_us":20837,"threads_started":1}
I20260619 15:03:36.027096 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.998100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37143 (local address 127.0.0.1:45524)
0619 15:03:36.015894 (+ 17794us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:36.015917 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:36.016036 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:36.017879 (+ 1843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:36.017883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:36.017906 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:36.018359 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.018373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.020583 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.020588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.024531 (+ 3943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.024546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.026160 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.026166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.026335 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.026382 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 15:03:36.026488 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 15:03:36.026634 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17630,"mutex_wait_us":26,"thread_start_us":12659,"threads_started":1}
I20260619 15:03:36.026685 32718 tablet_service.cc:1467] Processing CreateTablet for tablet 89731043fff741a9a0214af8aca5ea82 (DEFAULT_TABLE table=test-table [id=97e34539a92d482b8e0a1cf2f788fc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:36.027347 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.012085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.24.103.66:44825)
0619 15:03:36.019374 (+ 7289us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.019385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.019415 (+ 30us) server_negotiation.cc:407] Connection header received
0619 15:03:36.019569 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.019576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.019785 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.019977 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:36.020998 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.022455 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.024772 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.026222 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.026349 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.026524 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.026905 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.027078 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7155,"thread_start_us":7025,"threads_started":1}
I20260619 15:03:36.027928 619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:35.998578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.24.103.67:37143)
0619 15:03:36.013530 (+ 14952us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.013541 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.016198 (+ 2657us) server_negotiation.cc:407] Connection header received
0619 15:03:36.016308 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.016316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.016565 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.016777 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:36.018638 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.020403 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.024742 (+ 4339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.025929 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.026085 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.026336 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.027548 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.027712 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14755,"thread_start_us":19640,"threads_started":1}
I20260619 15:03:36.028337 616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.011763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:44825 (local address 127.0.0.1:37488)
0619 15:03:36.019026 (+ 7263us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:36.019050 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:36.019163 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:36.019964 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:36.019969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:36.019996 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:36.020835 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.020848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.022688 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.022695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.024589 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.024604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.026492 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.026499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.026674 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.026701 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 15:03:36.026837 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 15:03:36.026969 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7134,"mutex_wait_us":23,"thread_start_us":5260,"threads_started":1}
I20260619 15:03:36.028348 32718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89731043fff741a9a0214af8aca5ea82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:36.031697 528 tablet_service.cc:1467] Processing CreateTablet for tablet 89731043fff741a9a0214af8aca5ea82 (DEFAULT_TABLE table=test-table [id=97e34539a92d482b8e0a1cf2f788fc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:36.031697 389 tablet_service.cc:1467] Processing CreateTablet for tablet 89731043fff741a9a0214af8aca5ea82 (DEFAULT_TABLE table=test-table [id=97e34539a92d482b8e0a1cf2f788fc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:36.033254 528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89731043fff741a9a0214af8aca5ea82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:36.033284 389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89731043fff741a9a0214af8aca5ea82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:36.041280 622 tablet_bootstrap.cc:492] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906: Bootstrap starting.
I20260619 15:03:36.045991 622 tablet_bootstrap.cc:654] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:36.047387 622 log.cc:824] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:36.048410 623 tablet_bootstrap.cc:492] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a: Bootstrap starting.
I20260619 15:03:36.050242 625 tablet_bootstrap.cc:492] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a: Bootstrap starting.
I20260619 15:03:36.052465 623 tablet_bootstrap.cc:654] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:36.053316 625 tablet_bootstrap.cc:654] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:36.053581 622 tablet_bootstrap.cc:492] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906: No bootstrap required, opened a new log
I20260619 15:03:36.053795 623 log.cc:824] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:36.053892 622 ts_tablet_manager.cc:1397] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260619 15:03:36.054332 625 log.cc:824] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:36.057134 623 tablet_bootstrap.cc:492] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a: No bootstrap required, opened a new log
I20260619 15:03:36.057415 623 ts_tablet_manager.cc:1397] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 15:03:36.059430 625 tablet_bootstrap.cc:492] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a: No bootstrap required, opened a new log
I20260619 15:03:36.059796 625 ts_tablet_manager.cc:1397] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 15:03:36.065507 622 raft_consensus.cc:348] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.066057 622 raft_consensus.cc:374] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:36.066229 622 raft_consensus.cc:729] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf49c9af084419b9f46614f84987906, State: Initialized, Role: FOLLOWER
I20260619 15:03:36.066983 625 raft_consensus.cc:348] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.067003 622 consensus_queue.cc:260] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.067356 625 raft_consensus.cc:374] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:36.067483 625 raft_consensus.cc:729] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbc87acce8748a49111dd15c74e5f6a, State: Initialized, Role: FOLLOWER
I20260619 15:03:36.068243 625 consensus_queue.cc:260] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.069041 622 ts_tablet_manager.cc:1428] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260619 15:03:36.069275 459 heartbeater.cc:503] Master 127.24.103.126:41781 was elected leader, sending a full tablet report...
I20260619 15:03:36.069854 625 ts_tablet_manager.cc:1428] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260619 15:03:36.070992 623 raft_consensus.cc:348] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.071425 623 raft_consensus.cc:374] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:36.071523 623 raft_consensus.cc:729] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a821c4a3b474f25b5241f603dfb694a, State: Initialized, Role: FOLLOWER
I20260619 15:03:36.072057 623 consensus_queue.cc:260] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.074136 598 heartbeater.cc:503] Master 127.24.103.126:41781 was elected leader, sending a full tablet report...
I20260619 15:03:36.074631 623 ts_tablet_manager.cc:1428] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a: Time spent starting tablet: real 0.017s user 0.009s sys 0.005s
W20260619 15:03:36.084781 321 tablet.cc:2369] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:03:36.119658 460 tablet.cc:2369] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:03:36.133114 630 raft_consensus.cc:482] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:03:36.133392 630 raft_consensus.cc:504] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.134974 630 leader_election.cc:288] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495), 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825)
I20260619 15:03:36.141461 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.135565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:48583 (local address 127.24.103.66:44825)
0619 15:03:36.135753 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.135760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.135783 (+ 23us) server_negotiation.cc:407] Connection header received
0619 15:03:36.135999 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.136005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.136100 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.136212 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:36.136658 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.137605 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.139555 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.140663 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.140787 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.140838 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.141186 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.141275 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 15:03:36.141559 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.135297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:44825 (local address 127.24.103.67:48583)
0619 15:03:36.135632 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:36.135655 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:36.135813 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:36.136244 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:36.136247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:36.136273 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:36.136513 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.136524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.137735 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.137741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.139414 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.139425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.140821 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.140827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.141022 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.141057 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 15:03:36.141144 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 15:03:36.141277 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":20}
I20260619 15:03:36.142403 617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.135556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:34323 (local address 127.24.103.65:38495)
0619 15:03:36.135753 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.135760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.136401 (+ 641us) server_negotiation.cc:407] Connection header received
0619 15:03:36.136545 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.136551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.136653 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.136731 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:36.137282 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.138344 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.140443 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.141559 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.141712 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.141788 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.142131 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.142248 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:03:36.142550 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.135417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38495 (local address 127.24.103.67:34323)
0619 15:03:36.136281 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:36.136303 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:36.136424 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:36.136771 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:36.136775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:36.136797 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:36.137139 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.137150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.138459 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.138480 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.140296 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.140309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.141845 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.141854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.141972 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.141993 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 15:03:36.142070 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 15:03:36.142172 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":18,"thread_start_us":533,"threads_started":1}
I20260619 15:03:36.142725 414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89731043fff741a9a0214af8aca5ea82" candidate_uuid: "9a821c4a3b474f25b5241f603dfb694a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbc87acce8748a49111dd15c74e5f6a" is_pre_election: true
I20260619 15:03:36.143340 414 raft_consensus.cc:2454] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a821c4a3b474f25b5241f603dfb694a in term 0.
I20260619 15:03:36.144050 482 leader_election.cc:302] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fbc87acce8748a49111dd15c74e5f6a, 9a821c4a3b474f25b5241f603dfb694a; no voters:
I20260619 15:03:36.144177 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89731043fff741a9a0214af8aca5ea82" candidate_uuid: "9a821c4a3b474f25b5241f603dfb694a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf49c9af084419b9f46614f84987906" is_pre_election: true
I20260619 15:03:36.144516 630 raft_consensus.cc:2781] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:03:36.144732 630 raft_consensus.cc:482] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:03:36.144733 32743 raft_consensus.cc:2454] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a821c4a3b474f25b5241f603dfb694a in term 0.
I20260619 15:03:36.144850 630 raft_consensus.cc:3037] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:36.147159 630 raft_consensus.cc:504] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.147734 630 leader_election.cc:288] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [CANDIDATE]: Term 1 election: Requested vote from peers fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495), 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825)
I20260619 15:03:36.148114 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89731043fff741a9a0214af8aca5ea82" candidate_uuid: "9a821c4a3b474f25b5241f603dfb694a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf49c9af084419b9f46614f84987906"
I20260619 15:03:36.148298 32743 raft_consensus.cc:3037] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:36.150396 32743 raft_consensus.cc:2454] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a821c4a3b474f25b5241f603dfb694a in term 1.
I20260619 15:03:36.150693 414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89731043fff741a9a0214af8aca5ea82" candidate_uuid: "9a821c4a3b474f25b5241f603dfb694a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbc87acce8748a49111dd15c74e5f6a"
I20260619 15:03:36.150851 481 leader_election.cc:302] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a821c4a3b474f25b5241f603dfb694a, fdf49c9af084419b9f46614f84987906; no voters:
I20260619 15:03:36.150949 414 raft_consensus.cc:3037] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:36.151149 630 raft_consensus.cc:2781] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:36.151615 630 raft_consensus.cc:686] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 1 LEADER]: Becoming Leader. State: Replica: 9a821c4a3b474f25b5241f603dfb694a, State: Running, Role: LEADER
I20260619 15:03:36.152165 630 consensus_queue.cc:237] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
W20260619 15:03:36.153119 599 tablet.cc:2369] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:03:36.153810 414 raft_consensus.cc:2454] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a821c4a3b474f25b5241f603dfb694a in term 1.
I20260619 15:03:36.156802 32581 catalog_manager.cc:5495] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a821c4a3b474f25b5241f603dfb694a (127.24.103.67). New cstate: current_term: 1 leader_uuid: "9a821c4a3b474f25b5241f603dfb694a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } health_report { overall_health: UNKNOWN } } }
I20260619 15:03:36.187325 619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.181462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.24.103.67:37143)
0619 15:03:36.181667 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.181674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.181693 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:03:36.181799 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.181804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.181895 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.182048 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:03:36.182363 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.183516 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.184820 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.185473 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.185591 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.185981 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:03:36.186699 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:03:36.186900 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.186949 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.187042 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:03:36.196916 414 raft_consensus.cc:1261] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Refusing update from remote peer 9a821c4a3b474f25b5241f603dfb694a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:36.196916 32743 raft_consensus.cc:1261] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Refusing update from remote peer 9a821c4a3b474f25b5241f603dfb694a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:36.198036 632 consensus_queue.cc:1035] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:36.198421 630 consensus_queue.cc:1035] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:36.209987 635 mvcc.cc:205] Tried to move back new op lower bound from 7298586280731021312 to 7298586280563593216. Current Snapshot: MvccSnapshot[applied={T|T < 7298586280731021312}]
I20260619 15:03:36.215289 637 mvcc.cc:205] Tried to move back new op lower bound from 7298586280731021312 to 7298586280563593216. Current Snapshot: MvccSnapshot[applied={T|T < 7298586280731021312}]
I20260619 15:03:36.223486 32580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:36.223785 32580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:36.227617 32580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81: Sending DeleteTablet for 3 replicas of tablet 89731043fff741a9a0214af8aca5ea82
I20260619 15:03:36.229053 389 tablet_service.cc:1514] Processing DeleteTablet for tablet 89731043fff741a9a0214af8aca5ea82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:36 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:37488
I20260619 15:03:36.229053 32718 tablet_service.cc:1514] Processing DeleteTablet for tablet 89731043fff741a9a0214af8aca5ea82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:36 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:35866
I20260619 15:03:36.229323 528 tablet_service.cc:1514] Processing DeleteTablet for tablet 89731043fff741a9a0214af8aca5ea82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:36 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:45524
I20260619 15:03:36.233299 32581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:03:36.234161 32581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:03:36.241681 647 tablet_replica.cc:330] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a: stopping tablet replica
I20260619 15:03:36.243006 647 raft_consensus.cc:2229] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:03:36.243739 647 raft_consensus.cc:2258] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:36.244556 32717 tablet_service.cc:1467] Processing CreateTablet for tablet b7a33da5754048b58ba3f43d2b2ff296 (DEFAULT_TABLE table=test-table [id=bb269e4c49f9414aaa26069c2baf95ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:36.244876 528 tablet_service.cc:1467] Processing CreateTablet for tablet b7a33da5754048b58ba3f43d2b2ff296 (DEFAULT_TABLE table=test-table [id=bb269e4c49f9414aaa26069c2baf95ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:36.244867 388 tablet_service.cc:1467] Processing CreateTablet for tablet b7a33da5754048b58ba3f43d2b2ff296 (DEFAULT_TABLE table=test-table [id=bb269e4c49f9414aaa26069c2baf95ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:36.245160 32717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a33da5754048b58ba3f43d2b2ff296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:36.245460 388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a33da5754048b58ba3f43d2b2ff296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:36.245852 528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a33da5754048b58ba3f43d2b2ff296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:36.252270 623 tablet_bootstrap.cc:492] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a: Bootstrap starting.
I20260619 15:03:36.252297 625 tablet_bootstrap.cc:492] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a: Bootstrap starting.
I20260619 15:03:36.254043 623 tablet_bootstrap.cc:654] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:36.254036 625 tablet_bootstrap.cc:654] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:36.256783 623 tablet_bootstrap.cc:492] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a: No bootstrap required, opened a new log
I20260619 15:03:36.256944 623 ts_tablet_manager.cc:1397] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260619 15:03:36.257522 625 tablet_bootstrap.cc:492] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a: No bootstrap required, opened a new log
I20260619 15:03:36.257668 625 ts_tablet_manager.cc:1397] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 15:03:36.257858 623 raft_consensus.cc:348] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:36.258090 623 raft_consensus.cc:374] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:36.258198 623 raft_consensus.cc:729] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a821c4a3b474f25b5241f603dfb694a, State: Initialized, Role: FOLLOWER
I20260619 15:03:36.258468 625 raft_consensus.cc:348] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:36.258498 623 consensus_queue.cc:260] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:36.258694 625 raft_consensus.cc:374] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:36.258824 625 raft_consensus.cc:729] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbc87acce8748a49111dd15c74e5f6a, State: Initialized, Role: FOLLOWER
I20260619 15:03:36.259143 625 consensus_queue.cc:260] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:36.260203 623 ts_tablet_manager.cc:1428] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 15:03:36.261250 622 tablet_bootstrap.cc:492] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906: Bootstrap starting.
I20260619 15:03:36.263123 622 tablet_bootstrap.cc:654] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:36.266323 625 ts_tablet_manager.cc:1428] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20260619 15:03:36.277024 647 ts_tablet_manager.cc:1905] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:36.284308 645 tablet_replica.cc:330] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a: stopping tablet replica
I20260619 15:03:36.288671 622 tablet_bootstrap.cc:492] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906: No bootstrap required, opened a new log
I20260619 15:03:36.289044 622 ts_tablet_manager.cc:1397] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.016s
I20260619 15:03:36.285025 645 raft_consensus.cc:2229] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:36.290086 622 raft_consensus.cc:348] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:36.290408 622 raft_consensus.cc:374] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:36.290563 622 raft_consensus.cc:729] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf49c9af084419b9f46614f84987906, State: Initialized, Role: FOLLOWER
I20260619 15:03:36.290642 645 raft_consensus.cc:2258] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:36.290973 622 consensus_queue.cc:260] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:36.292315 622 ts_tablet_manager.cc:1428] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 15:03:36.293411 646 tablet_replica.cc:330] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906: stopping tablet replica
I20260619 15:03:36.293882 646 raft_consensus.cc:2229] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:36.294442 646 raft_consensus.cc:2258] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:36.296353 645 ts_tablet_manager.cc:1905] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:36.297737 647 ts_tablet_manager.cc:1918] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:36.297924 647 log.cc:1196] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/89731043fff741a9a0214af8aca5ea82
I20260619 15:03:36.298385 647 ts_tablet_manager.cc:1939] T 89731043fff741a9a0214af8aca5ea82 P 9a821c4a3b474f25b5241f603dfb694a: Deleting consensus metadata
I20260619 15:03:36.300138 32568 catalog_manager.cc:4841] TS 9a821c4a3b474f25b5241f603dfb694a (127.24.103.67:37143): tablet 89731043fff741a9a0214af8aca5ea82 (table test-table [id=97e34539a92d482b8e0a1cf2f788fc4c]) successfully deleted
I20260619 15:03:36.301767 646 ts_tablet_manager.cc:1905] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:36.302536 645 ts_tablet_manager.cc:1918] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:36.302706 645 log.cc:1196] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/89731043fff741a9a0214af8aca5ea82
I20260619 15:03:36.303269 645 ts_tablet_manager.cc:1939] T 89731043fff741a9a0214af8aca5ea82 P 7fbc87acce8748a49111dd15c74e5f6a: Deleting consensus metadata
I20260619 15:03:36.305056 32568 catalog_manager.cc:4841] TS 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825): tablet 89731043fff741a9a0214af8aca5ea82 (table test-table [id=97e34539a92d482b8e0a1cf2f788fc4c]) successfully deleted
I20260619 15:03:36.306972 646 ts_tablet_manager.cc:1918] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:36.307145 646 log.cc:1196] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/89731043fff741a9a0214af8aca5ea82
I20260619 15:03:36.307591 646 ts_tablet_manager.cc:1939] T 89731043fff741a9a0214af8aca5ea82 P fdf49c9af084419b9f46614f84987906: Deleting consensus metadata
I20260619 15:03:36.309001 32567 catalog_manager.cc:4841] TS fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495): tablet 89731043fff741a9a0214af8aca5ea82 (table test-table [id=97e34539a92d482b8e0a1cf2f788fc4c]) successfully deleted
I20260619 15:03:36.383723 628 raft_consensus.cc:482] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:03:36.383991 628 raft_consensus.cc:504] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:36.385727 628 leader_election.cc:288] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825), 9a821c4a3b474f25b5241f603dfb694a (127.24.103.67:37143)
I20260619 15:03:36.393429 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.386073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:58559 (local address 127.24.103.66:44825)
0619 15:03:36.386255 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.386262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.386609 (+ 347us) server_negotiation.cc:407] Connection header received
0619 15:03:36.386929 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.386935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.387024 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.387105 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:36.387988 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.389119 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.391116 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.392285 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.392459 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.392534 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.392991 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.393125 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 15:03:36.393502 651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.385945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:44825 (local address 127.24.103.65:58559)
0619 15:03:36.386494 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:36.386516 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:36.386723 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:36.387322 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:36.387326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:36.387358 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:36.387756 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.387775 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.389284 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.389290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.390969 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.390980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.392535 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.392542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.392759 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.392800 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 15:03:36.392940 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 15:03:36.393108 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":489,"threads_started":1}
I20260619 15:03:36.394340 414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a33da5754048b58ba3f43d2b2ff296" candidate_uuid: "fdf49c9af084419b9f46614f84987906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbc87acce8748a49111dd15c74e5f6a" is_pre_election: true
I20260619 15:03:36.394629 414 raft_consensus.cc:2454] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdf49c9af084419b9f46614f84987906 in term 0.
I20260619 15:03:36.395133 32671 leader_election.cc:302] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fbc87acce8748a49111dd15c74e5f6a, fdf49c9af084419b9f46614f84987906; no voters:
I20260619 15:03:36.395579 628 raft_consensus.cc:2781] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:03:36.395720 628 raft_consensus.cc:482] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:03:36.395817 628 raft_consensus.cc:3037] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:36.396584 619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.387114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:48499 (local address 127.24.103.67:37143)
0619 15:03:36.387300 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.387305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.388750 (+ 1445us) server_negotiation.cc:407] Connection header received
0619 15:03:36.388942 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.388948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.389016 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.389137 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:36.389974 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.391275 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.393913 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.395284 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.395440 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.395522 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.396129 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.396255 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:03:36.396639 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.387427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37143 (local address 127.24.103.65:48499)
0619 15:03:36.388612 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:36.388631 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:36.388754 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:36.389243 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:36.389248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:36.389276 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:36.389740 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.389752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.391416 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.391423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.393364 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.393378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.395718 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.395724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.395928 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.395961 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 15:03:36.396091 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 15:03:36.396236 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":852,"threads_started":1}
I20260619 15:03:36.397588 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a33da5754048b58ba3f43d2b2ff296" candidate_uuid: "fdf49c9af084419b9f46614f84987906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a821c4a3b474f25b5241f603dfb694a" is_pre_election: true
I20260619 15:03:36.397965 548 raft_consensus.cc:2454] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdf49c9af084419b9f46614f84987906 in term 0.
I20260619 15:03:36.399206 628 raft_consensus.cc:504] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:36.399943 628 leader_election.cc:288] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [CANDIDATE]: Term 1 election: Requested vote from peers 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825), 9a821c4a3b474f25b5241f603dfb694a (127.24.103.67:37143)
I20260619 15:03:36.400455 414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a33da5754048b58ba3f43d2b2ff296" candidate_uuid: "fdf49c9af084419b9f46614f84987906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbc87acce8748a49111dd15c74e5f6a"
I20260619 15:03:36.400516 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a33da5754048b58ba3f43d2b2ff296" candidate_uuid: "fdf49c9af084419b9f46614f84987906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a821c4a3b474f25b5241f603dfb694a"
I20260619 15:03:36.400713 548 raft_consensus.cc:3037] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:36.400704 414 raft_consensus.cc:3037] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:36.403852 414 raft_consensus.cc:2454] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdf49c9af084419b9f46614f84987906 in term 1.
I20260619 15:03:36.404083 548 raft_consensus.cc:2454] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdf49c9af084419b9f46614f84987906 in term 1.
I20260619 15:03:36.404471 32671 leader_election.cc:302] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fbc87acce8748a49111dd15c74e5f6a, fdf49c9af084419b9f46614f84987906; no voters:
I20260619 15:03:36.404807 628 raft_consensus.cc:2781] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:36.405298 628 raft_consensus.cc:686] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 1 LEADER]: Becoming Leader. State: Replica: fdf49c9af084419b9f46614f84987906, State: Running, Role: LEADER
I20260619 15:03:36.405735 628 consensus_queue.cc:237] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:36.409706 32581 catalog_manager.cc:5495] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdf49c9af084419b9f46614f84987906 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "fdf49c9af084419b9f46614f84987906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } health_report { overall_health: UNKNOWN } } }
Jun 19 15:03:36 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881414, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.126@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:36.465123 660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.445589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41781 (local address 127.0.0.1:55238)
0619 15:03:36.446270 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:36.446298 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:36.446434 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:36.446845 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:36.446848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:36.447514 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:36.447797 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.447806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.449329 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.449338 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.450479 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.450492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.451572 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.451577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.451714 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.455967 (+ 4253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:36.456000 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:36.458598 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:36.462530 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:36.462543 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:36.462549 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:36.463123 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:36.463510 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:36.463514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:36.463517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:36.463673 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:36.464269 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:36.464275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:36.464433 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 15:03:36.464565 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 15:03:36.464803 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":369,"threads_started":1}
I20260619 15:03:36.465384 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.445492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.24.103.126:41781)
0619 15:03:36.445817 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.445826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.446411 (+ 585us) server_negotiation.cc:407] Connection header received
0619 15:03:36.446556 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.446561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.446640 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.446755 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:36.447955 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.449137 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.450620 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.451341 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.451474 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.458883 (+ 7409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:36.458893 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:36.458900 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:36.458941 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:36.462354 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:36.463241 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:36.463246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:36.463250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:36.463382 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:36.463789 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:36.463793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:36.463797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:36.464148 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:36.464298 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.464694 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.464967 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":143}
I20260619 15:03:36.470885 32581 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:55238:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 15:03:36.472019 32581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:03:36.479128 528 tablet_service.cc:1467] Processing CreateTablet for tablet f5e03cc1dd2e465684ad5c4236cfeb5e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b80e3fb34fb4cb5bccd5f71f8eb7741]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 15:03:36.479126 32718 tablet_service.cc:1467] Processing CreateTablet for tablet f5e03cc1dd2e465684ad5c4236cfeb5e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b80e3fb34fb4cb5bccd5f71f8eb7741]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 15:03:36.479776 528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e03cc1dd2e465684ad5c4236cfeb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:36.479770 32718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e03cc1dd2e465684ad5c4236cfeb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:36.487394 389 tablet_service.cc:1467] Processing CreateTablet for tablet f5e03cc1dd2e465684ad5c4236cfeb5e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b80e3fb34fb4cb5bccd5f71f8eb7741]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 15:03:36.487778 623 tablet_bootstrap.cc:492] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: Bootstrap starting.
I20260619 15:03:36.488023 389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e03cc1dd2e465684ad5c4236cfeb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:36.489658 623 tablet_bootstrap.cc:654] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:36.492290 623 tablet_bootstrap.cc:492] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: No bootstrap required, opened a new log
I20260619 15:03:36.492476 623 ts_tablet_manager.cc:1397] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260619 15:03:36.493353 623 raft_consensus.cc:348] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.493638 623 raft_consensus.cc:374] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:36.493728 623 raft_consensus.cc:729] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a821c4a3b474f25b5241f603dfb694a, State: Initialized, Role: FOLLOWER
I20260619 15:03:36.494006 623 consensus_queue.cc:260] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.494324 625 tablet_bootstrap.cc:492] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a: Bootstrap starting.
I20260619 15:03:36.494442 642 tablet_replica.cc:439] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } }
I20260619 15:03:36.495216 622 tablet_bootstrap.cc:492] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906: Bootstrap starting.
I20260619 15:03:36.495788 642 tablet_replica.cc:442] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:03:36.496068 625 tablet_bootstrap.cc:654] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:36.496933 622 tablet_bootstrap.cc:654] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:36.498183 623 ts_tablet_manager.cc:1428] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260619 15:03:36.502732 625 tablet_bootstrap.cc:492] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a: No bootstrap required, opened a new log
I20260619 15:03:36.502899 622 tablet_bootstrap.cc:492] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906: No bootstrap required, opened a new log
I20260619 15:03:36.503047 625 ts_tablet_manager.cc:1397] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260619 15:03:36.503082 622 ts_tablet_manager.cc:1397] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260619 15:03:36.503865 625 raft_consensus.cc:348] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.503855 622 raft_consensus.cc:348] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.504112 622 raft_consensus.cc:374] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:36.504105 625 raft_consensus.cc:374] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:36.504180 622 raft_consensus.cc:729] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf49c9af084419b9f46614f84987906, State: Initialized, Role: FOLLOWER
I20260619 15:03:36.504184 625 raft_consensus.cc:729] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbc87acce8748a49111dd15c74e5f6a, State: Initialized, Role: FOLLOWER
I20260619 15:03:36.504455 622 consensus_queue.cc:260] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.504479 625 consensus_queue.cc:260] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.504901 628 tablet_replica.cc:439] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } }
I20260619 15:03:36.505131 625 ts_tablet_manager.cc:1428] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 15:03:36.505208 628 tablet_replica.cc:442] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:03:36.504971 629 tablet_replica.cc:439] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } }
I20260619 15:03:36.505332 629 tablet_replica.cc:442] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:03:36.506635 622 ts_tablet_manager.cc:1428] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 15:03:36.656894 642 raft_consensus.cc:482] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:03:36.657136 642 raft_consensus.cc:504] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.658039 642 leader_election.cc:288] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495), 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825)
I20260619 15:03:36.658628 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e03cc1dd2e465684ad5c4236cfeb5e" candidate_uuid: "9a821c4a3b474f25b5241f603dfb694a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf49c9af084419b9f46614f84987906" is_pre_election: true
I20260619 15:03:36.658630 414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e03cc1dd2e465684ad5c4236cfeb5e" candidate_uuid: "9a821c4a3b474f25b5241f603dfb694a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbc87acce8748a49111dd15c74e5f6a" is_pre_election: true
I20260619 15:03:36.658944 32743 raft_consensus.cc:2454] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a821c4a3b474f25b5241f603dfb694a in term 0.
I20260619 15:03:36.658942 414 raft_consensus.cc:2454] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a821c4a3b474f25b5241f603dfb694a in term 0.
I20260619 15:03:36.659466 482 leader_election.cc:302] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fbc87acce8748a49111dd15c74e5f6a, 9a821c4a3b474f25b5241f603dfb694a; no voters:
I20260619 15:03:36.659807 642 raft_consensus.cc:2781] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:03:36.659950 642 raft_consensus.cc:482] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:03:36.660053 642 raft_consensus.cc:3037] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:36.663048 642 raft_consensus.cc:504] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.663762 642 leader_election.cc:288] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [CANDIDATE]: Term 1 election: Requested vote from peers fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495), 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825)
I20260619 15:03:36.664310 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e03cc1dd2e465684ad5c4236cfeb5e" candidate_uuid: "9a821c4a3b474f25b5241f603dfb694a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf49c9af084419b9f46614f84987906"
I20260619 15:03:36.664314 414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e03cc1dd2e465684ad5c4236cfeb5e" candidate_uuid: "9a821c4a3b474f25b5241f603dfb694a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbc87acce8748a49111dd15c74e5f6a"
I20260619 15:03:36.664533 32743 raft_consensus.cc:3037] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:36.664547 414 raft_consensus.cc:3037] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:36.666625 32743 raft_consensus.cc:2454] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a821c4a3b474f25b5241f603dfb694a in term 1.
I20260619 15:03:36.667202 481 leader_election.cc:302] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a821c4a3b474f25b5241f603dfb694a, fdf49c9af084419b9f46614f84987906; no voters:
I20260619 15:03:36.667517 642 raft_consensus.cc:2781] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:36.667701 642 raft_consensus.cc:686] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [term 1 LEADER]: Becoming Leader. State: Replica: 9a821c4a3b474f25b5241f603dfb694a, State: Running, Role: LEADER
I20260619 15:03:36.667762 414 raft_consensus.cc:2454] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a821c4a3b474f25b5241f603dfb694a in term 1.
I20260619 15:03:36.667969 642 consensus_queue.cc:237] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } }
I20260619 15:03:36.669611 630 tablet_replica.cc:439] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: TxnStatusTablet state changed. Reason: New leader 9a821c4a3b474f25b5241f603dfb694a. Latest consensus state: current_term: 1 leader_uuid: "9a821c4a3b474f25b5241f603dfb694a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } }
I20260619 15:03:36.669927 630 tablet_replica.cc:442] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: This TxnStatusTablet replica's current role is: LEADER
I20260619 15:03:36.670650 665 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 15:03:36.670943 665 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 15:03:36.671463 32581 catalog_manager.cc:5495] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a821c4a3b474f25b5241f603dfb694a (127.24.103.67). New cstate: current_term: 1 leader_uuid: "9a821c4a3b474f25b5241f603dfb694a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } health_report { overall_health: UNKNOWN } } }
I20260619 15:03:36.703500 619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.697609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.24.103.67:37143)
0619 15:03:36.698096 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.698105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.698120 (+ 15us) server_negotiation.cc:407] Connection header received
0619 15:03:36.698173 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.698176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.698233 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.698309 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:03:36.698745 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.699584 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.701643 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.702219 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.702330 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.702898 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:03:36.703004 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:03:36.703073 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.703244 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.703306 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386}
I20260619 15:03:36.703704 660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.697493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37143 (local address 127.0.0.1:45112)
0619 15:03:36.697752 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:36.697768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:36.697879 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:36.698379 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:36.698384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:36.698398 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 15:03:36.698600 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.698611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.699718 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.699724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.701427 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.701445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.702384 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.702390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.702585 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.702663 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 15:03:36.703108 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 15:03:36.703129 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 15:03:36.703220 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 15:03:36.703387 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 15:03:36.707448 414 raft_consensus.cc:1261] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Refusing update from remote peer 9a821c4a3b474f25b5241f603dfb694a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:36.707469 32743 raft_consensus.cc:1261] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Refusing update from remote peer 9a821c4a3b474f25b5241f603dfb694a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:36.708084 642 consensus_queue.cc:1035] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:36.708367 630 consensus_queue.cc:1035] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:36.711035 629 tablet_replica.cc:439] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a: TxnStatusTablet state changed. Reason: New leader 9a821c4a3b474f25b5241f603dfb694a. Latest consensus state: current_term: 1 leader_uuid: "9a821c4a3b474f25b5241f603dfb694a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } }
I20260619 15:03:36.711424 629 tablet_replica.cc:442] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:03:36.712237 628 tablet_replica.cc:439] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906: TxnStatusTablet state changed. Reason: New leader 9a821c4a3b474f25b5241f603dfb694a. Latest consensus state: current_term: 1 leader_uuid: "9a821c4a3b474f25b5241f603dfb694a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } }
I20260619 15:03:36.712529 628 tablet_replica.cc:442] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:03:36.713806 630 tablet_replica.cc:439] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a821c4a3b474f25b5241f603dfb694a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } }
I20260619 15:03:36.714102 630 tablet_replica.cc:442] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: This TxnStatusTablet replica's current role is: LEADER
I20260619 15:03:36.715062 632 tablet_replica.cc:439] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a821c4a3b474f25b5241f603dfb694a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } }
I20260619 15:03:36.715440 632 tablet_replica.cc:442] T f5e03cc1dd2e465684ad5c4236cfeb5e P 9a821c4a3b474f25b5241f603dfb694a: This TxnStatusTablet replica's current role is: LEADER
I20260619 15:03:36.717175 629 tablet_replica.cc:439] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a821c4a3b474f25b5241f603dfb694a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } }
I20260619 15:03:36.717417 629 tablet_replica.cc:442] T f5e03cc1dd2e465684ad5c4236cfeb5e P 7fbc87acce8748a49111dd15c74e5f6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:03:36.719413 628 tablet_replica.cc:439] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a821c4a3b474f25b5241f603dfb694a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } }
I20260619 15:03:36.719728 628 tablet_replica.cc:442] T f5e03cc1dd2e465684ad5c4236cfeb5e P fdf49c9af084419b9f46614f84987906: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 15:03:36.730726 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.724318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.24.103.65:38495)
0619 15:03:36.725246 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.725250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.725267 (+ 17us) server_negotiation.cc:407] Connection header received
0619 15:03:36.725346 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.725349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.725420 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.725591 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:03:36.726089 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.727016 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.728402 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.728822 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.728959 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.729423 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:03:36.729983 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:03:36.730148 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.730260 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.730342 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":662,"threads_started":1}
I20260619 15:03:36.748443 619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.741796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:58465 (local address 127.24.103.67:37143)
0619 15:03:36.741972 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.741977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.743302 (+ 1325us) server_negotiation.cc:407] Connection header received
0619 15:03:36.743522 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.743535 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.743608 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.743703 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:03:36.744226 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.745045 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.746529 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.747079 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.747213 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.747644 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:03:36.747771 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:03:36.747866 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.747998 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.748094 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 15:03:36.748550 675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.741702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37143 (local address 127.24.103.65:58465)
0619 15:03:36.743195 (+ 1493us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:36.743211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:36.743383 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:36.743718 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:36.743723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:36.743734 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 15:03:36.744064 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.744086 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.745172 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.745177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.746385 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.746398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.747257 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.747264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.747394 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.747458 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 15:03:36.747868 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 15:03:36.747881 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 15:03:36.747978 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 15:03:36.748144 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1350,"thread_start_us":1229,"threads_started":1}
I20260619 15:03:36.759080 414 raft_consensus.cc:1261] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Refusing update from remote peer fdf49c9af084419b9f46614f84987906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:36.759317 548 raft_consensus.cc:1261] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Refusing update from remote peer fdf49c9af084419b9f46614f84987906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:36.759886 628 consensus_queue.cc:1035] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:36.760278 653 consensus_queue.cc:1035] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:36.807651 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.799032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:42689 (local address 127.24.103.65:38495)
0619 15:03:36.799212 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.799219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.801027 (+ 1808us) server_negotiation.cc:407] Connection header received
0619 15:03:36.801172 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.801178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.801277 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.801424 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:03:36.801965 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.803132 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.805290 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.805992 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.806101 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.806888 (+ 787us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:03:36.807049 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:03:36.807120 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:36.807362 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 15:03:36.807433 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 15:03:36.808000 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.799052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38495 (local address 127.24.103.67:42689)
0619 15:03:36.800922 (+ 1870us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:36.800940 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:36.801052 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:36.801451 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:36.801455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:36.801467 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 15:03:36.801804 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.801817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.803313 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.803320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.805108 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:36.805125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.806313 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.806320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:36.806556 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.806641 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 15:03:36.807218 (+ 577us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 15:03:36.807229 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 15:03:36.807321 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 15:03:36.807494 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":703,"threads_started":1}
I20260619 15:03:36.855571 32581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:36.855885 32581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:36.860037 32581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81: Sending DeleteTablet for 3 replicas of tablet b7a33da5754048b58ba3f43d2b2ff296
I20260619 15:03:36.861227 389 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a33da5754048b58ba3f43d2b2ff296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:36 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:37488
I20260619 15:03:36.861533 528 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a33da5754048b58ba3f43d2b2ff296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:36 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:45524
I20260619 15:03:36.861678 32718 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a33da5754048b58ba3f43d2b2ff296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:36 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:35866
I20260619 15:03:36.897835 686 tablet_replica.cc:330] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906: stopping tablet replica
I20260619 15:03:36.898175 686 raft_consensus.cc:2229] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:03:36.898881 686 raft_consensus.cc:2258] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:36.900750 684 tablet_replica.cc:330] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a: stopping tablet replica
I20260619 15:03:36.901032 684 raft_consensus.cc:2229] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:36.901328 684 raft_consensus.cc:2258] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:36.903185 684 ts_tablet_manager.cc:1905] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:36.906045 686 ts_tablet_manager.cc:1905] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:36.915293 685 tablet_replica.cc:330] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a: stopping tablet replica
I20260619 15:03:36.915685 685 raft_consensus.cc:2229] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:36.916097 685 raft_consensus.cc:2258] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:36.917429 685 ts_tablet_manager.cc:1905] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:36.910696 684 ts_tablet_manager.cc:1918] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:03:36.923754 684 log.cc:1196] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/b7a33da5754048b58ba3f43d2b2ff296
I20260619 15:03:36.923774 686 ts_tablet_manager.cc:1918] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:03:36.924105 686 log.cc:1196] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/b7a33da5754048b58ba3f43d2b2ff296
I20260619 15:03:36.924350 684 ts_tablet_manager.cc:1939] T b7a33da5754048b58ba3f43d2b2ff296 P 7fbc87acce8748a49111dd15c74e5f6a: Deleting consensus metadata
I20260619 15:03:36.925698 32568 catalog_manager.cc:4841] TS 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825): tablet b7a33da5754048b58ba3f43d2b2ff296 (table test-table [id=bb269e4c49f9414aaa26069c2baf95ee]) successfully deleted
I20260619 15:03:36.931622 686 ts_tablet_manager.cc:1939] T b7a33da5754048b58ba3f43d2b2ff296 P fdf49c9af084419b9f46614f84987906: Deleting consensus metadata
I20260619 15:03:36.931901 685 ts_tablet_manager.cc:1918] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:03:36.932152 685 log.cc:1196] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/b7a33da5754048b58ba3f43d2b2ff296
I20260619 15:03:36.932770 685 ts_tablet_manager.cc:1939] T b7a33da5754048b58ba3f43d2b2ff296 P 9a821c4a3b474f25b5241f603dfb694a: Deleting consensus metadata
I20260619 15:03:36.933354 32567 catalog_manager.cc:4841] TS fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495): tablet b7a33da5754048b58ba3f43d2b2ff296 (table test-table [id=bb269e4c49f9414aaa26069c2baf95ee]) successfully deleted
I20260619 15:03:36.934012 32568 catalog_manager.cc:4841] TS 9a821c4a3b474f25b5241f603dfb694a (127.24.103.67:37143): tablet b7a33da5754048b58ba3f43d2b2ff296 (table test-table [id=bb269e4c49f9414aaa26069c2baf95ee]) successfully deleted
Jun 19 15:03:36 dist-test-slave-zpbw krb5kdc[32536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 15:03:36 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:36 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 15:03:36.983456 693 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:36.969071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.24.103.126:41781)
0619 15:03:36.969566 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.969573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.970121 (+ 548us) server_negotiation.cc:407] Connection header received
0619 15:03:36.970335 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.970342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.970444 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.970595 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:36.972438 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.973703 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.975783 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.976494 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.976644 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:36.983063 (+ 6419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55248: BlockingRecv error: recv got EOF from 127.0.0.1:55248 (error 108)
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":266,"threads_started":1}
Jun 19 15:03:36 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260619 15:03:37.007977 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:36.987369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.24.103.126:41781)
0619 15:03:36.987570 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:36.987576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:36.988211 (+ 635us) server_negotiation.cc:407] Connection header received
0619 15:03:36.988400 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:36.988406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:36.988523 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:36.988634 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:36.990308 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.991566 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.993726 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:36.994509 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:36.994661 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.000140 (+ 5479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:37.000148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:37.000153 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:37.000206 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:37.004127 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:37.005107 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:37.005113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:37.005117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:37.005284 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:37.005809 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:37.005816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:37.005819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:37.006232 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:37.006522 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:37.007205 (+ 683us) server_negotiation.cc:299] Negotiation successful
0619 15:03:37.007495 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:03:37.012208 32581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:03:37.012965 32581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:03:37.021416 32718 tablet_service.cc:1467] Processing CreateTablet for tablet 4073b9fd07fa43da89321461434af957 (DEFAULT_TABLE table=test-table [id=2f7ed2616bcd446e96327258753a9cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:37.021699 389 tablet_service.cc:1467] Processing CreateTablet for tablet 4073b9fd07fa43da89321461434af957 (DEFAULT_TABLE table=test-table [id=2f7ed2616bcd446e96327258753a9cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:37.021806 528 tablet_service.cc:1467] Processing CreateTablet for tablet 4073b9fd07fa43da89321461434af957 (DEFAULT_TABLE table=test-table [id=2f7ed2616bcd446e96327258753a9cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:37.021941 32718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4073b9fd07fa43da89321461434af957. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:37.022122 389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4073b9fd07fa43da89321461434af957. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:37.022244 528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4073b9fd07fa43da89321461434af957. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:37.036893 702 tablet_bootstrap.cc:492] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a: Bootstrap starting.
I20260619 15:03:37.043324 701 tablet_bootstrap.cc:492] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a: Bootstrap starting.
I20260619 15:03:37.045125 701 tablet_bootstrap.cc:654] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:37.045125 700 tablet_bootstrap.cc:492] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906: Bootstrap starting.
I20260619 15:03:37.046939 700 tablet_bootstrap.cc:654] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:37.048282 701 tablet_bootstrap.cc:492] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a: No bootstrap required, opened a new log
I20260619 15:03:37.048545 701 ts_tablet_manager.cc:1397] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 15:03:37.049474 701 raft_consensus.cc:348] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.049907 702 tablet_bootstrap.cc:654] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:37.049891 701 raft_consensus.cc:374] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:37.050079 701 raft_consensus.cc:729] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a821c4a3b474f25b5241f603dfb694a, State: Initialized, Role: FOLLOWER
I20260619 15:03:37.050410 701 consensus_queue.cc:260] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.052281 701 ts_tablet_manager.cc:1428] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 15:03:37.056749 700 tablet_bootstrap.cc:492] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906: No bootstrap required, opened a new log
I20260619 15:03:37.056929 700 ts_tablet_manager.cc:1397] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20260619 15:03:37.057825 700 raft_consensus.cc:348] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.058101 700 raft_consensus.cc:374] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:37.058211 700 raft_consensus.cc:729] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf49c9af084419b9f46614f84987906, State: Initialized, Role: FOLLOWER
I20260619 15:03:37.058459 700 consensus_queue.cc:260] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.059092 700 ts_tablet_manager.cc:1428] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 15:03:37.060601 702 tablet_bootstrap.cc:492] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a: No bootstrap required, opened a new log
I20260619 15:03:37.060819 702 ts_tablet_manager.cc:1397] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20260619 15:03:37.061725 702 raft_consensus.cc:348] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.062104 702 raft_consensus.cc:374] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:37.062247 702 raft_consensus.cc:729] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbc87acce8748a49111dd15c74e5f6a, State: Initialized, Role: FOLLOWER
I20260619 15:03:37.062565 702 consensus_queue.cc:260] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.063230 702 ts_tablet_manager.cc:1428] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 15:03:37.451586 642 raft_consensus.cc:482] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:03:37.451838 642 raft_consensus.cc:504] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.452539 642 leader_election.cc:288] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495), 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825)
I20260619 15:03:37.453153 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4073b9fd07fa43da89321461434af957" candidate_uuid: "9a821c4a3b474f25b5241f603dfb694a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf49c9af084419b9f46614f84987906" is_pre_election: true
I20260619 15:03:37.453176 414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4073b9fd07fa43da89321461434af957" candidate_uuid: "9a821c4a3b474f25b5241f603dfb694a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbc87acce8748a49111dd15c74e5f6a" is_pre_election: true
I20260619 15:03:37.453382 32743 raft_consensus.cc:2454] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a821c4a3b474f25b5241f603dfb694a in term 0.
I20260619 15:03:37.453447 414 raft_consensus.cc:2454] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a821c4a3b474f25b5241f603dfb694a in term 0.
I20260619 15:03:37.453929 481 leader_election.cc:302] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a821c4a3b474f25b5241f603dfb694a, fdf49c9af084419b9f46614f84987906; no voters:
I20260619 15:03:37.454196 642 raft_consensus.cc:2781] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:03:37.454331 642 raft_consensus.cc:482] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:03:37.454444 642 raft_consensus.cc:3037] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:37.456545 642 raft_consensus.cc:504] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.457162 642 leader_election.cc:288] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [CANDIDATE]: Term 1 election: Requested vote from peers fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495), 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825)
I20260619 15:03:37.457682 414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4073b9fd07fa43da89321461434af957" candidate_uuid: "9a821c4a3b474f25b5241f603dfb694a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbc87acce8748a49111dd15c74e5f6a"
I20260619 15:03:37.457687 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4073b9fd07fa43da89321461434af957" candidate_uuid: "9a821c4a3b474f25b5241f603dfb694a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf49c9af084419b9f46614f84987906"
I20260619 15:03:37.457886 414 raft_consensus.cc:3037] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:37.457904 32743 raft_consensus.cc:3037] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:37.459951 414 raft_consensus.cc:2454] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a821c4a3b474f25b5241f603dfb694a in term 1.
I20260619 15:03:37.460045 32743 raft_consensus.cc:2454] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a821c4a3b474f25b5241f603dfb694a in term 1.
I20260619 15:03:37.460503 482 leader_election.cc:302] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fbc87acce8748a49111dd15c74e5f6a, 9a821c4a3b474f25b5241f603dfb694a; no voters:
I20260619 15:03:37.460815 642 raft_consensus.cc:2781] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:37.461158 642 raft_consensus.cc:686] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 1 LEADER]: Becoming Leader. State: Replica: 9a821c4a3b474f25b5241f603dfb694a, State: Running, Role: LEADER
I20260619 15:03:37.461449 642 consensus_queue.cc:237] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.464231 32579 catalog_manager.cc:5495] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a821c4a3b474f25b5241f603dfb694a (127.24.103.67). New cstate: current_term: 1 leader_uuid: "9a821c4a3b474f25b5241f603dfb694a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } health_report { overall_health: HEALTHY } } }
I20260619 15:03:37.503762 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:37.499022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.24.103.67:37143)
0619 15:03:37.499552 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:37.499557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:37.499574 (+ 17us) server_negotiation.cc:407] Connection header received
0619 15:03:37.499634 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:37.499638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:37.499700 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:37.499784 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:03:37.500144 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.500843 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.502093 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.502484 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.502573 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.503007 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:03:37.503115 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:03:37.503182 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:37.503308 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 15:03:37.503371 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":292,"threads_started":1}
I20260619 15:03:37.506655 414 raft_consensus.cc:1261] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Refusing update from remote peer 9a821c4a3b474f25b5241f603dfb694a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:37.506656 32743 raft_consensus.cc:1261] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Refusing update from remote peer 9a821c4a3b474f25b5241f603dfb694a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:37.507349 706 consensus_queue.cc:1035] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:37.507632 642 consensus_queue.cc:1035] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:37.514940 712 mvcc.cc:205] Tried to move back new op lower bound from 7298586286101504000 to 7298586285924872192. Current Snapshot: MvccSnapshot[applied={T|T < 7298586286101504000}]
I20260619 15:03:37.526098 32579 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:37.526326 32579 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:37.529206 32579 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81: Sending DeleteTablet for 3 replicas of tablet 4073b9fd07fa43da89321461434af957
I20260619 15:03:37.530045 32718 tablet_service.cc:1514] Processing DeleteTablet for tablet 4073b9fd07fa43da89321461434af957 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:37 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:35866
I20260619 15:03:37.530061 389 tablet_service.cc:1514] Processing DeleteTablet for tablet 4073b9fd07fa43da89321461434af957 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:37 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:37488
I20260619 15:03:37.530308 528 tablet_service.cc:1514] Processing DeleteTablet for tablet 4073b9fd07fa43da89321461434af957 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:37 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:45524
I20260619 15:03:37.531359 32579 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:03:37.532307 32579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:03:37.541273 32717 tablet_service.cc:1467] Processing CreateTablet for tablet 71c3158b53d0454f9e46645158b11c3a (DEFAULT_TABLE table=test-table [id=b507827881294de8a13244f080fe9639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:37.541455 527 tablet_service.cc:1467] Processing CreateTablet for tablet 71c3158b53d0454f9e46645158b11c3a (DEFAULT_TABLE table=test-table [id=b507827881294de8a13244f080fe9639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:37.541455 388 tablet_service.cc:1467] Processing CreateTablet for tablet 71c3158b53d0454f9e46645158b11c3a (DEFAULT_TABLE table=test-table [id=b507827881294de8a13244f080fe9639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:03:37.541898 388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c3158b53d0454f9e46645158b11c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:37.541919 527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c3158b53d0454f9e46645158b11c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:37.541918 32717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c3158b53d0454f9e46645158b11c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:37.546048 718 tablet_replica.cc:330] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906: stopping tablet replica
I20260619 15:03:37.546269 718 raft_consensus.cc:2229] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:37.546542 718 raft_consensus.cc:2258] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:37.547150 719 tablet_replica.cc:330] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a: stopping tablet replica
I20260619 15:03:37.547358 719 raft_consensus.cc:2229] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:37.547650 719 raft_consensus.cc:2258] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:37.548725 701 tablet_bootstrap.cc:492] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a: Bootstrap starting.
I20260619 15:03:37.548959 718 ts_tablet_manager.cc:1905] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:37.550127 701 tablet_bootstrap.cc:654] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:37.552481 702 tablet_bootstrap.cc:492] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a: Bootstrap starting.
I20260619 15:03:37.553647 700 tablet_bootstrap.cc:492] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906: Bootstrap starting.
I20260619 15:03:37.554729 702 tablet_bootstrap.cc:654] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:37.555413 700 tablet_bootstrap.cc:654] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:37.555742 718 ts_tablet_manager.cc:1918] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:37.555887 718 log.cc:1196] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/4073b9fd07fa43da89321461434af957
I20260619 15:03:37.556361 718 ts_tablet_manager.cc:1939] T 4073b9fd07fa43da89321461434af957 P fdf49c9af084419b9f46614f84987906: Deleting consensus metadata
I20260619 15:03:37.557510 32567 catalog_manager.cc:4841] TS fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495): tablet 4073b9fd07fa43da89321461434af957 (table test-table [id=2f7ed2616bcd446e96327258753a9cf0]) successfully deleted
I20260619 15:03:37.560726 719 ts_tablet_manager.cc:1905] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:37.560856 720 tablet_replica.cc:330] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a: stopping tablet replica
I20260619 15:03:37.561062 720 raft_consensus.cc:2229] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:03:37.561471 720 raft_consensus.cc:2258] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:37.561655 701 tablet_bootstrap.cc:492] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a: No bootstrap required, opened a new log
I20260619 15:03:37.561820 701 ts_tablet_manager.cc:1397] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20260619 15:03:37.562572 701 raft_consensus.cc:348] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.563081 701 raft_consensus.cc:374] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:37.563181 701 raft_consensus.cc:729] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a821c4a3b474f25b5241f603dfb694a, State: Initialized, Role: FOLLOWER
I20260619 15:03:37.563411 701 consensus_queue.cc:260] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.563926 701 ts_tablet_manager.cc:1428] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 15:03:37.568536 700 tablet_bootstrap.cc:492] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906: No bootstrap required, opened a new log
I20260619 15:03:37.568822 700 ts_tablet_manager.cc:1397] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.002s
I20260619 15:03:37.569231 719 ts_tablet_manager.cc:1918] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:37.569442 719 log.cc:1196] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/4073b9fd07fa43da89321461434af957
I20260619 15:03:37.570004 719 ts_tablet_manager.cc:1939] T 4073b9fd07fa43da89321461434af957 P 7fbc87acce8748a49111dd15c74e5f6a: Deleting consensus metadata
I20260619 15:03:37.569875 700 raft_consensus.cc:348] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.570120 700 raft_consensus.cc:374] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:37.570200 700 raft_consensus.cc:729] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf49c9af084419b9f46614f84987906, State: Initialized, Role: FOLLOWER
I20260619 15:03:37.570433 700 consensus_queue.cc:260] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.571034 700 ts_tablet_manager.cc:1428] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 15:03:37.571142 720 ts_tablet_manager.cc:1905] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:37.572378 32568 catalog_manager.cc:4841] TS 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825): tablet 4073b9fd07fa43da89321461434af957 (table test-table [id=2f7ed2616bcd446e96327258753a9cf0]) successfully deleted
I20260619 15:03:37.572551 702 tablet_bootstrap.cc:492] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a: No bootstrap required, opened a new log
I20260619 15:03:37.572731 702 ts_tablet_manager.cc:1397] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.010s
I20260619 15:03:37.573551 702 raft_consensus.cc:348] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.573819 702 raft_consensus.cc:374] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:37.573892 702 raft_consensus.cc:729] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbc87acce8748a49111dd15c74e5f6a, State: Initialized, Role: FOLLOWER
I20260619 15:03:37.574118 702 consensus_queue.cc:260] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.574698 702 ts_tablet_manager.cc:1428] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 15:03:37.576319 720 ts_tablet_manager.cc:1918] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:03:37.576395 629 raft_consensus.cc:482] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:03:37.576505 720 log.cc:1196] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/4073b9fd07fa43da89321461434af957
I20260619 15:03:37.576575 629 raft_consensus.cc:504] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.577013 720 ts_tablet_manager.cc:1939] T 4073b9fd07fa43da89321461434af957 P 9a821c4a3b474f25b5241f603dfb694a: Deleting consensus metadata
I20260619 15:03:37.578085 32568 catalog_manager.cc:4841] TS 9a821c4a3b474f25b5241f603dfb694a (127.24.103.67:37143): tablet 4073b9fd07fa43da89321461434af957 (table test-table [id=2f7ed2616bcd446e96327258753a9cf0]) successfully deleted
I20260619 15:03:37.578418 629 leader_election.cc:288] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495), 9a821c4a3b474f25b5241f603dfb694a (127.24.103.67:37143)
I20260619 15:03:37.584981 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:37.579036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:54935 (local address 127.24.103.67:37143)
0619 15:03:37.579307 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:37.579315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:37.579335 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:03:37.579580 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:37.579584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:37.579639 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:37.579739 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:37.580242 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.581002 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.582876 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.583907 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.584083 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.584163 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:37.584553 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 15:03:37.584693 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260619 15:03:37.585083 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:37.578689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37143 (local address 127.24.103.66:54935)
0619 15:03:37.579255 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:37.579274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:37.579403 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:37.579754 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:37.579758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:37.579782 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:37.580087 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:37.580099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.581162 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.581168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:37.582673 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:37.582685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.584118 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.584125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:37.584371 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.584406 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 15:03:37.584501 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 15:03:37.584679 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":290,"threads_started":1}
I20260619 15:03:37.585754 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c3158b53d0454f9e46645158b11c3a" candidate_uuid: "7fbc87acce8748a49111dd15c74e5f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a821c4a3b474f25b5241f603dfb694a" is_pre_election: true
I20260619 15:03:37.586095 548 raft_consensus.cc:2454] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fbc87acce8748a49111dd15c74e5f6a in term 0.
I20260619 15:03:37.586679 343 leader_election.cc:302] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fbc87acce8748a49111dd15c74e5f6a, 9a821c4a3b474f25b5241f603dfb694a; no voters:
I20260619 15:03:37.587173 629 raft_consensus.cc:2781] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:03:37.587339 629 raft_consensus.cc:482] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:03:37.587460 629 raft_consensus.cc:3037] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:37.591123 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:37.578866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:38495 (local address 127.24.103.66:40169)
0619 15:03:37.585449 (+ 6583us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:37.585469 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:37.585587 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:37.585934 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:37.585939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:37.585961 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:03:37.586145 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:37.586152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.587550 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.587564 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:37.589072 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:37.589084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.590258 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.590265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:37.590486 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.590530 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 15:03:37.590615 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 15:03:37.590755 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6453,"threads_started":1}
I20260619 15:03:37.591259 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:37.579515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:40169 (local address 127.24.103.65:38495)
0619 15:03:37.580087 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:37.580096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:37.585689 (+ 5593us) server_negotiation.cc:407] Connection header received
0619 15:03:37.585751 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:37.585754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:37.585820 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:37.585923 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:03:37.586306 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.587380 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.589275 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.590076 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.590673 (+ 597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.590754 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:37.590832 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 15:03:37.590964 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":294,"threads_started":1}
I20260619 15:03:37.592159 629 raft_consensus.cc:504] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.592576 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c3158b53d0454f9e46645158b11c3a" candidate_uuid: "7fbc87acce8748a49111dd15c74e5f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf49c9af084419b9f46614f84987906" is_pre_election: true
I20260619 15:03:37.592823 32743 raft_consensus.cc:2454] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fbc87acce8748a49111dd15c74e5f6a in term 0.
I20260619 15:03:37.593480 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c3158b53d0454f9e46645158b11c3a" candidate_uuid: "7fbc87acce8748a49111dd15c74e5f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a821c4a3b474f25b5241f603dfb694a"
I20260619 15:03:37.593595 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c3158b53d0454f9e46645158b11c3a" candidate_uuid: "7fbc87acce8748a49111dd15c74e5f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf49c9af084419b9f46614f84987906"
I20260619 15:03:37.593709 548 raft_consensus.cc:3037] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:37.593791 32743 raft_consensus.cc:3037] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:37.594897 629 leader_election.cc:288] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [CANDIDATE]: Term 1 election: Requested vote from peers fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495), 9a821c4a3b474f25b5241f603dfb694a (127.24.103.67:37143)
I20260619 15:03:37.595984 32743 raft_consensus.cc:2454] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fbc87acce8748a49111dd15c74e5f6a in term 1.
I20260619 15:03:37.596004 548 raft_consensus.cc:2454] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fbc87acce8748a49111dd15c74e5f6a in term 1.
I20260619 15:03:37.596482 342 leader_election.cc:302] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7fbc87acce8748a49111dd15c74e5f6a, 9a821c4a3b474f25b5241f603dfb694a, fdf49c9af084419b9f46614f84987906; no voters:
I20260619 15:03:37.596825 629 raft_consensus.cc:2781] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:37.597209 629 raft_consensus.cc:686] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 1 LEADER]: Becoming Leader. State: Replica: 7fbc87acce8748a49111dd15c74e5f6a, State: Running, Role: LEADER
I20260619 15:03:37.597586 629 consensus_queue.cc:237] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } }
I20260619 15:03:37.601035 32581 catalog_manager.cc:5495] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66). New cstate: current_term: 1 leader_uuid: "7fbc87acce8748a49111dd15c74e5f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fbc87acce8748a49111dd15c74e5f6a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 } health_report { overall_health: UNKNOWN } } }
I20260619 15:03:37.620914 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:37.615190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.24.103.66:44825)
0619 15:03:37.615668 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:37.615672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:37.615690 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:03:37.615769 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:37.615773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:37.615845 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:37.615987 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:03:37.616398 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.617244 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.618572 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.619071 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.619168 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.619643 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:03:37.620255 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:03:37.620371 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:37.620451 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 15:03:37.620525 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":262,"threads_started":1}
I20260619 15:03:37.635950 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:37.631072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:46237 (local address 127.24.103.67:37143)
0619 15:03:37.631221 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:37.631225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:37.631477 (+ 252us) server_negotiation.cc:407] Connection header received
0619 15:03:37.631630 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:37.631634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:37.631696 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:37.631778 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:03:37.632229 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.632980 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.634315 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.634718 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.634883 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.635305 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:03:37.635458 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:03:37.635526 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:37.635661 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 15:03:37.635731 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 15:03:37.636005 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:37.630704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37143 (local address 127.24.103.66:46237)
0619 15:03:37.631392 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:37.631408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:37.631524 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:37.631809 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:37.631812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:37.631821 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 15:03:37.632099 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:37.632107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.633093 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.633096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:37.634188 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:37.634199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.634914 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.634919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:37.635027 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.635084 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 15:03:37.635546 (+ 462us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 15:03:37.635555 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 15:03:37.635628 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 15:03:37.635731 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":373,"threads_started":1}
I20260619 15:03:37.644836 548 raft_consensus.cc:1261] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Refusing update from remote peer 7fbc87acce8748a49111dd15c74e5f6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:37.644836 32743 raft_consensus.cc:1261] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Refusing update from remote peer 7fbc87acce8748a49111dd15c74e5f6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:03:37.645512 727 consensus_queue.cc:1035] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf49c9af084419b9f46614f84987906" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 38495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:37.645870 629 consensus_queue.cc:1035] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a821c4a3b474f25b5241f603dfb694a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:03:37.670949 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:37.665757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:50369 (local address 127.24.103.66:44825)
0619 15:03:37.665963 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:37.665973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:37.666347 (+ 374us) server_negotiation.cc:407] Connection header received
0619 15:03:37.666475 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:37.666478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:37.666539 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:37.666616 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:03:37.667053 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.667839 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.669401 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.669885 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.669982 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.670319 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:03:37.670412 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:03:37.670475 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:37.670592 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 15:03:37.670654 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 15:03:37.671061 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:37.665696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:44825 (local address 127.24.103.67:50369)
0619 15:03:37.666253 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:37.666269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:37.666377 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:37.666643 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:37.666646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:37.666654 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 15:03:37.666933 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:37.666943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.667973 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.667978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:37.669241 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:37.669255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.670019 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.670023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:37.670153 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.670189 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 15:03:37.670490 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 15:03:37.670498 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 15:03:37.670556 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 15:03:37.670660 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":275,"threads_started":1}
I20260619 15:03:37.712106 32581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:37.712361 32581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:03:37.715607 32581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea9c434a3d6b4dcd84b0852cf7f7dc81: Sending DeleteTablet for 3 replicas of tablet 71c3158b53d0454f9e46645158b11c3a
I20260619 15:03:37.716565 32717 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c3158b53d0454f9e46645158b11c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:37 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:35866
I20260619 15:03:37.716734 389 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c3158b53d0454f9e46645158b11c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:37 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:37488
I20260619 15:03:37.716864 718 tablet_replica.cc:330] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906: stopping tablet replica
I20260619 15:03:37.717018 719 tablet_replica.cc:330] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a: stopping tablet replica
I20260619 15:03:37.717064 718 raft_consensus.cc:2229] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:37.717233 719 raft_consensus.cc:2229] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:03:37.717278 718 raft_consensus.cc:2258] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:37.717739 719 raft_consensus.cc:2258] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:37.718056 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 32652
I20260619 15:03:37.718847 718 ts_tablet_manager.cc:1905] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:37.718925 719 ts_tablet_manager.cc:1905] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:37.722890 528 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c3158b53d0454f9e46645158b11c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:03:37 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:45524
I20260619 15:03:37.723397 720 tablet_replica.cc:330] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a: stopping tablet replica
I20260619 15:03:37.723672 720 raft_consensus.cc:2229] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:03:37.723869 720 raft_consensus.cc:2258] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:03:37.725069 720 ts_tablet_manager.cc:1905] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:03:37.726827 718 ts_tablet_manager.cc:1918] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:03:37.727034 718 log.cc:1196] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/71c3158b53d0454f9e46645158b11c3a
I20260619 15:03:37.727578 718 ts_tablet_manager.cc:1939] T 71c3158b53d0454f9e46645158b11c3a P fdf49c9af084419b9f46614f84987906: Deleting consensus metadata
I20260619 15:03:37.727770 719 ts_tablet_manager.cc:1918] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:03:37.727967 719 log.cc:1196] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/71c3158b53d0454f9e46645158b11c3a
I20260619 15:03:37.728495 719 ts_tablet_manager.cc:1939] T 71c3158b53d0454f9e46645158b11c3a P 7fbc87acce8748a49111dd15c74e5f6a: Deleting consensus metadata
I20260619 15:03:37.729156 32567 catalog_manager.cc:4841] TS fdf49c9af084419b9f46614f84987906 (127.24.103.65:38495): tablet 71c3158b53d0454f9e46645158b11c3a (table test-table [id=b507827881294de8a13244f080fe9639]) successfully deleted
I20260619 15:03:37.729826 32568 catalog_manager.cc:4841] TS 7fbc87acce8748a49111dd15c74e5f6a (127.24.103.66:44825): tablet 71c3158b53d0454f9e46645158b11c3a (table test-table [id=b507827881294de8a13244f080fe9639]) successfully deleted
I20260619 15:03:37.731827 720 ts_tablet_manager.cc:1918] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 15:03:37.732002 720 log.cc:1196] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/71c3158b53d0454f9e46645158b11c3a
I20260619 15:03:37.732424 720 ts_tablet_manager.cc:1939] T 71c3158b53d0454f9e46645158b11c3a P 9a821c4a3b474f25b5241f603dfb694a: Deleting consensus metadata
I20260619 15:03:37.733402 32568 catalog_manager.cc:4841] TS 9a821c4a3b474f25b5241f603dfb694a (127.24.103.67:37143): tablet 71c3158b53d0454f9e46645158b11c3a (table test-table [id=b507827881294de8a13244f080fe9639]) successfully deleted
Jun 19 15:03:37 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.65@KRBTEST.COM
I20260619 15:03:37.746905 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:37.719805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60364 (local address 127.24.103.65:38495)
0619 15:03:37.719974 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:37.719980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:37.730099 (+ 10119us) server_negotiation.cc:407] Connection header received
0619 15:03:37.730397 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:37.730403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:37.730486 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:37.730602 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:37.732297 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.733424 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.735419 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.735828 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.735931 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.740884 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:37.740901 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:37.740916 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:37.740954 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:37.743882 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:37.744687 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:37.744694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:37.744697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:37.744817 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:37.745283 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:37.745288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:37.745290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:37.745660 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:37.745953 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:37.746248 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 15:03:37.746462 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 15:03:37.747526 315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:37.873287 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 32652
I20260619 15:03:37.901667 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 324
Jun 19 15:03:37 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.66@KRBTEST.COM
I20260619 15:03:37.922308 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:37.902474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.24.103.66:44825)
0619 15:03:37.903098 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:37.903104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:37.903121 (+ 17us) server_negotiation.cc:407] Connection header received
0619 15:03:37.903179 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:37.903185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:37.903296 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:37.903409 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:37.905082 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.906227 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.908134 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:37.908825 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:37.908941 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:37.914419 (+ 5478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:37.914435 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:37.914460 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:37.914509 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:37.918537 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:37.919527 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:37.919537 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:37.919540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:37.919702 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:37.920130 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:37.920135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:37.920139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:37.920596 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:37.920913 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:37.921246 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 15:03:37.921541 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477}
I20260619 15:03:37.923095 454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:38.046103 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 324
I20260619 15:03:38.065335 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 463
Jun 19 15:03:38 dist-test-slave-zpbw krb5kdc[32536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.67@KRBTEST.COM
I20260619 15:03:38.079970 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:38.066137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.24.103.67:37143)
0619 15:03:38.066312 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:38.066316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:38.066334 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:03:38.066468 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:38.066471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:38.066529 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:38.066627 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:38.067918 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:38.068559 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:38.069759 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:38.070168 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:38.070259 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:38.074596 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:38.074613 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:38.074628 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:38.074664 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:38.077354 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:38.078110 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:38.078117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:38.078119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:38.078249 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:38.078561 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:38.078564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:38.078565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:38.078965 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:38.079188 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:38.079549 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 15:03:38.079728 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 15:03:38.080703 593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:38.223887 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 463
I20260619 15:03:38.249621 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 32549
I20260619 15:03:38.250878 32621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:38.365419 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 32549
2026-06-19T15:03:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5968 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:03:38 dist-test-slave-zpbw krb5kdc[749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:03:38 dist-test-slave-zpbw krb5kdc[749](info): set up 2 sockets
Jun 19 15:03:38 dist-test-slave-zpbw krb5kdc[749](info): commencing operation
krb5kdc: starting...
W20260619 15:03:39.468248 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:03:39 dist-test-slave-zpbw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:03:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:03:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:03:39.694847 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43617
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:43617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:40.047142 762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:40.047452 762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:40.047541 762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:40.054847 762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:40.054976 762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:40.055022 762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:40.055071 762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:40.055140 762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:03:40.065167 762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:43617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:40.066558 762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:40.067823 762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:40.080073 770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:40.080083 768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:40.080273 767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:40.080783 762 server_base.cc:1034] running on GCE node
I20260619 15:03:40.081689 762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:40.083253 762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:40.084456 762 hybrid_clock.cc:648] HybridClock initialized: now 1781881420084388 us; error 54 us; skew 500 ppm
I20260619 15:03:40.084766 762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T15:03:40Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1788 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:03:40 dist-test-slave-zpbw krb5kdc[776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:03:40 dist-test-slave-zpbw krb5kdc[776](info): set up 2 sockets
Jun 19 15:03:40 dist-test-slave-zpbw krb5kdc[776](info): commencing operation
krb5kdc: starting...
W20260619 15:03:41.248142 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:03:41 dist-test-slave-zpbw krb5kdc[776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:03:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:03:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:03:41.474238 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34275
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:34275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:41.808977 789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:41.809298 789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:41.809476 789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:41.817039 789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:41.817163 789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:41.817291 789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:41.817391 789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:41.817469 789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:03:41.827701 789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:34275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:41.829118 789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:41.830461 789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:41.848145 794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:41.848109 797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:41.848615 795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:41.849506 789 server_base.cc:1034] running on GCE node
I20260619 15:03:41.850425 789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:41.852160 789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:41.853395 789 hybrid_clock.cc:648] HybridClock initialized: now 1781881421853343 us; error 48 us; skew 500 ppm
I20260619 15:03:41.853749 789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:41 dist-test-slave-zpbw krb5kdc[776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881421, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:41.862555 789 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:03:41.864348 789 webserver.cc:460] Webserver started at http://127.24.103.126:43333/ using document root <none> and password file <none>
I20260619 15:03:41.865124 789 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:41.865275 789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:41.865630 789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:41.868855 789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "f1ac2fca7cd54d94a25d87314cb2a97c"
format_stamp: "Formatted at 2026-06-19 15:03:41 on dist-test-slave-zpbw"
server_key: "e8bff2a4eddd2189f45fe63a3a83ad88"
server_key_iv: "9e13b926483c0b3d766679ab750ae339"
server_key_version: "encryptionkey@0"
I20260619 15:03:41.869669 789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "f1ac2fca7cd54d94a25d87314cb2a97c"
format_stamp: "Formatted at 2026-06-19 15:03:41 on dist-test-slave-zpbw"
server_key: "e8bff2a4eddd2189f45fe63a3a83ad88"
server_key_iv: "9e13b926483c0b3d766679ab750ae339"
server_key_version: "encryptionkey@0"
I20260619 15:03:41.875631 789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 15:03:41.879324 804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:41.880755 789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 15:03:41.880908 789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "f1ac2fca7cd54d94a25d87314cb2a97c"
format_stamp: "Formatted at 2026-06-19 15:03:41 on dist-test-slave-zpbw"
server_key: "e8bff2a4eddd2189f45fe63a3a83ad88"
server_key_iv: "9e13b926483c0b3d766679ab750ae339"
server_key_version: "encryptionkey@0"
I20260619 15:03:41.881089 789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:41.901980 789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:41.906056 789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:41.906308 789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:41.925483 789 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:34275
I20260619 15:03:41.925509 855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:34275 every 8 connection(s)
I20260619 15:03:41.927040 789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:41.930953 856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:03:41.937106 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 789
I20260619 15:03:41.937335 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:03:41.938151 24989 external_mini_cluster.cc:1428] Setting key c295d88ec7f70ba3de75cc1010a987a2
I20260619 15:03:41.943041 856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c: Bootstrap starting.
I20260619 15:03:41.947127 856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c: Neither blocks nor log segments found. Creating new log.
I20260619 15:03:41.948370 856 log.cc:824] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:41.951838 856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c: No bootstrap required, opened a new log
Jun 19 15:03:41 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:41.960511 856 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34275 } }
I20260619 15:03:41.960945 856 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:03:41.961086 856 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ac2fca7cd54d94a25d87314cb2a97c, State: Initialized, Role: FOLLOWER
I20260619 15:03:41.961823 856 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34275 } }
I20260619 15:03:41.962095 856 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:41.962245 856 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:41.962433 856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:03:41.962915 859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:41.940557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.24.103.126:34275)
0619 15:03:41.941456 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:41.941469 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:41.941513 (+ 44us) server_negotiation.cc:407] Connection header received
0619 15:03:41.943378 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:41.943403 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:41.943832 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:41.944182 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:41.946149 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:41.947707 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:41.949495 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:41.950279 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:41.950411 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:41.955145 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:41.955153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:41.955170 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:41.955214 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:41.958354 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:41.959200 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:41.959207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:41.959214 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:41.959392 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:41.959782 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:41.959789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:41.959793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:41.960369 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:41.960703 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:41.961152 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 15:03:41.961636 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":470,"threads_started":1}
I20260619 15:03:41.965448 856 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34275 } }
I20260619 15:03:41.966120 856 leader_election.cc:302] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1ac2fca7cd54d94a25d87314cb2a97c; no voters:
I20260619 15:03:41.966724 856 leader_election.cc:288] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:03:41.966902 861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:03:41.967628 861 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 1 LEADER]: Becoming Leader. State: Replica: f1ac2fca7cd54d94a25d87314cb2a97c, State: Running, Role: LEADER
I20260619 15:03:41.968160 861 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34275 } }
I20260619 15:03:41.968956 856 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:41.971112 862 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34275 } } }
I20260619 15:03:41.971635 862 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:41.972304 863 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1ac2fca7cd54d94a25d87314cb2a97c. Latest consensus state: current_term: 1 leader_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34275 } } }
I20260619 15:03:41.972666 863 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:41.976198 868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:41.981942 868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:41.991067 868 catalog_manager.cc:1347] Generated new cluster ID: d2e9ffd417cc40cd8a3664f8770a9f93
I20260619 15:03:41.991194 868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:41.998446 868 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:03:42.000442 868 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:42.010195 868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c: Generated new TSK 0
I20260619 15:03:42.011166 868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:03:42.076218 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34275
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:42.424834 882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:42.425137 882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:42.425235 882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:42.432536 882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:42.432663 882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:42.432783 882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:03:42.443375 882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:42.444698 882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:42.445989 882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:42.459712 887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:42.459815 888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:42.459864 890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:42.461090 882 server_base.cc:1034] running on GCE node
I20260619 15:03:42.461776 882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:42.462833 882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:42.464097 882 hybrid_clock.cc:648] HybridClock initialized: now 1781881422464039 us; error 53 us; skew 500 ppm
I20260619 15:03:42.464455 882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:42 dist-test-slave-zpbw krb5kdc[776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881422, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:42.467677 882 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:03:42.469296 882 webserver.cc:460] Webserver started at http://127.24.103.65:45095/ using document root <none> and password file <none>
I20260619 15:03:42.470084 882 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:42.470239 882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:42.470556 882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:42.473604 882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "04413a52e2af4acb9ff263187ce33842"
format_stamp: "Formatted at 2026-06-19 15:03:42 on dist-test-slave-zpbw"
server_key: "028486490233981306a7f4125fdd3b40"
server_key_iv: "d614a7849b8b3051cb72e2e9de1bba9f"
server_key_version: "encryptionkey@0"
I20260619 15:03:42.474423 882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "04413a52e2af4acb9ff263187ce33842"
format_stamp: "Formatted at 2026-06-19 15:03:42 on dist-test-slave-zpbw"
server_key: "028486490233981306a7f4125fdd3b40"
server_key_iv: "d614a7849b8b3051cb72e2e9de1bba9f"
server_key_version: "encryptionkey@0"
I20260619 15:03:42.479844 882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 15:03:42.483521 897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:42.485014 882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:03:42.485204 882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "04413a52e2af4acb9ff263187ce33842"
format_stamp: "Formatted at 2026-06-19 15:03:42 on dist-test-slave-zpbw"
server_key: "028486490233981306a7f4125fdd3b40"
server_key_iv: "d614a7849b8b3051cb72e2e9de1bba9f"
server_key_version: "encryptionkey@0"
I20260619 15:03:42.485425 882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:42.506317 882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:42.511034 882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:42.511349 882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:42.512317 882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:42.513700 882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:42.513809 882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:42.513957 882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:42.514029 882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:42.547545 882 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:37319
I20260619 15:03:42.547600 1009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:37319 every 8 connection(s)
I20260619 15:03:42.548882 882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:03:42.552938 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 882
I20260619 15:03:42.553151 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:03:42.553793 24989 external_mini_cluster.cc:1428] Setting key 28aeac632819b2392c8dde3875f7116a
Jun 19 15:03:42 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881422, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:42.572259 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:42.551506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:44081 (local address 127.24.103.126:34275)
0619 15:03:42.551960 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:42.551965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:42.552986 (+ 1021us) server_negotiation.cc:407] Connection header received
0619 15:03:42.555189 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:42.555195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:42.555291 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:42.555414 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:42.557574 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:42.558307 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:42.560820 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:42.561222 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:42.561324 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:42.565855 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:42.565861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:42.565865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:42.565901 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:42.568617 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:42.569658 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:42.569662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:42.569664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:42.569790 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:42.570385 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:42.570391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:42.570398 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:42.570687 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:42.570860 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:42.571810 (+ 950us) server_negotiation.cc:299] Negotiation successful
0619 15:03:42.571990 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":233,"threads_started":1}
I20260619 15:03:42.574043 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:42.552112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:44081)
0619 15:03:42.552819 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:42.552863 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:42.554897 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:42.555670 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:42.555679 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:42.556444 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:42.557357 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:42.557389 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:42.558465 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:42.558473 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:42.560664 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:42.560684 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:42.561467 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:42.561477 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:42.561822 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:42.562618 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:42.562660 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:42.565576 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:42.568789 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:42.568802 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:42.568825 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:42.569517 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:42.569934 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:42.569942 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:42.569949 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:42.570214 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:42.570839 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:42.570851 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:42.571392 (+ 541us) client_negotiation.cc:769] Sending connection context
0619 15:03:42.571714 (+ 322us) client_negotiation.cc:240] Negotiation successful
0619 15:03:42.572350 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":297,"threads_started":1}
I20260619 15:03:42.576120 1010 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34275
I20260619 15:03:42.576454 1010 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:42.577277 1010 heartbeater.cc:511] Master 127.24.103.126:34275 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260619 15:03:42.579563 821 ts_manager.cc:194] Registered new tserver with Master: 04413a52e2af4acb9ff263187ce33842 (127.24.103.65:37319)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
I20260619 15:03:42.581350 821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:44081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:03:42.606014 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34275
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:42.949713 1016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:42.950014 1016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:42.950112 1016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:42.957603 1016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:42.957726 1016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:42.957842 1016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:03:42.967935 1016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:42.969185 1016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:42.970623 1016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:42.984022 1022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:42.984083 1021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:42.984299 1024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:42.989982 1016 server_base.cc:1034] running on GCE node
I20260619 15:03:42.990731 1016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:42.991737 1016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:42.993001 1016 hybrid_clock.cc:648] HybridClock initialized: now 1781881422992948 us; error 41 us; skew 500 ppm
I20260619 15:03:42.993364 1016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:42 dist-test-slave-zpbw krb5kdc[776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881422, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:42.996635 1016 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:03:42.998210 1016 webserver.cc:460] Webserver started at http://127.24.103.66:39577/ using document root <none> and password file <none>
I20260619 15:03:42.999013 1016 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:42.999154 1016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:42.999471 1016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:43.002370 1016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "220bfe4aa72945cc8c1c66c2d019507a"
format_stamp: "Formatted at 2026-06-19 15:03:42 on dist-test-slave-zpbw"
server_key: "b4f5e8f152ef70841ec73d189c4aa63c"
server_key_iv: "b79dc08e5afeb5d0b94379c1cce8c4ef"
server_key_version: "encryptionkey@0"
I20260619 15:03:43.003204 1016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "220bfe4aa72945cc8c1c66c2d019507a"
format_stamp: "Formatted at 2026-06-19 15:03:42 on dist-test-slave-zpbw"
server_key: "b4f5e8f152ef70841ec73d189c4aa63c"
server_key_iv: "b79dc08e5afeb5d0b94379c1cce8c4ef"
server_key_version: "encryptionkey@0"
I20260619 15:03:43.008263 1016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 15:03:43.011770 1031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:43.013070 1016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 15:03:43.013257 1016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "220bfe4aa72945cc8c1c66c2d019507a"
format_stamp: "Formatted at 2026-06-19 15:03:42 on dist-test-slave-zpbw"
server_key: "b4f5e8f152ef70841ec73d189c4aa63c"
server_key_iv: "b79dc08e5afeb5d0b94379c1cce8c4ef"
server_key_version: "encryptionkey@0"
I20260619 15:03:43.013453 1016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:43.046713 1016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:43.051020 1016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:43.051318 1016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:43.052279 1016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:43.053586 1016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:43.053718 1016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:43.053875 1016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:43.053947 1016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:43.087632 1016 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:38637
I20260619 15:03:43.087682 1143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:38637 every 8 connection(s)
I20260619 15:03:43.089104 1016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:03:43.099400 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 1016
I20260619 15:03:43.099584 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:03:43.100196 24989 external_mini_cluster.cc:1428] Setting key 9edfc2db78c55aae34ed1732b6608c16
Jun 19 15:03:43 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881422, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:43.109790 1146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:43.091705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:39777 (local address 127.24.103.126:34275)
0619 15:03:43.092127 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:43.092132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:43.093184 (+ 1052us) server_negotiation.cc:407] Connection header received
0619 15:03:43.095020 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:43.095025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:43.095103 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:43.095217 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:43.096735 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:43.097485 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:43.099203 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:43.099725 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:43.099821 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:43.103880 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:43.103886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:43.103889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:43.103921 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:43.106625 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:43.107429 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:43.107439 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:43.107441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:43.107576 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:43.108028 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:43.108032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:43.108034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:43.108309 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:43.108477 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:43.109195 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 15:03:43.109424 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":207,"threads_started":1}
I20260619 15:03:43.110857 1147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:43.092327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:39777)
0619 15:03:43.093012 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:43.093056 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:43.094686 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:43.095433 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:43.095441 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:43.095981 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:43.096555 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:43.096572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:43.097629 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:43.097632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:43.099054 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:43.099071 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:43.099959 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:43.099966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:43.100274 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:43.101060 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:43.101083 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:43.103630 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:43.106814 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:43.106824 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:43.106843 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:43.107288 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:43.107691 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:43.107696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:43.107700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:43.107897 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:43.108440 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:43.108448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:43.108821 (+ 373us) client_negotiation.cc:769] Sending connection context
0619 15:03:43.109090 (+ 269us) client_negotiation.cc:240] Negotiation successful
0619 15:03:43.109647 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":293,"threads_started":1}
I20260619 15:03:43.112309 1144 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34275
I20260619 15:03:43.112643 1144 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:43.113487 1144 heartbeater.cc:511] Master 127.24.103.126:34275 requested a full tablet report, sending...
I20260619 15:03:43.115176 821 ts_manager.cc:194] Registered new tserver with Master: 220bfe4aa72945cc8c1c66c2d019507a (127.24.103.66:38637)
I20260619 15:03:43.116478 821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:39777
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:03:43.148002 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34275
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:43.476032 1150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:43.476332 1150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:43.476516 1150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:43.483911 1150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:43.484037 1150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:43.484265 1150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:03:43.494410 1150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:43.495988 1150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:43.497310 1150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:43.510706 1155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:43.510598 1158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:43.510583 1156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:43.511559 1150 server_base.cc:1034] running on GCE node
I20260619 15:03:43.512267 1150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:43.513259 1150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:43.514492 1150 hybrid_clock.cc:648] HybridClock initialized: now 1781881423514434 us; error 55 us; skew 500 ppm
I20260619 15:03:43.514884 1150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:43 dist-test-slave-zpbw krb5kdc[776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881423, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:43.518141 1150 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:03:43.519665 1150 webserver.cc:460] Webserver started at http://127.24.103.67:45475/ using document root <none> and password file <none>
I20260619 15:03:43.520403 1150 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:43.520545 1150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:43.520849 1150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:03:43.523918 1150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "d9eedaf434f241e09203060b7cada11d"
format_stamp: "Formatted at 2026-06-19 15:03:43 on dist-test-slave-zpbw"
server_key: "a4828babadef951c47adf914aecac9b5"
server_key_iv: "8d7c9396ed58f7eda08c80e1978b1c69"
server_key_version: "encryptionkey@0"
I20260619 15:03:43.524744 1150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "d9eedaf434f241e09203060b7cada11d"
format_stamp: "Formatted at 2026-06-19 15:03:43 on dist-test-slave-zpbw"
server_key: "a4828babadef951c47adf914aecac9b5"
server_key_iv: "8d7c9396ed58f7eda08c80e1978b1c69"
server_key_version: "encryptionkey@0"
I20260619 15:03:43.529990 1150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 15:03:43.533771 1165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:43.535349 1150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 15:03:43.535527 1150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "d9eedaf434f241e09203060b7cada11d"
format_stamp: "Formatted at 2026-06-19 15:03:43 on dist-test-slave-zpbw"
server_key: "a4828babadef951c47adf914aecac9b5"
server_key_iv: "8d7c9396ed58f7eda08c80e1978b1c69"
server_key_version: "encryptionkey@0"
I20260619 15:03:43.535727 1150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:43.548031 1150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:43.552963 1150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:43.553293 1150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:43.554311 1150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:43.555838 1150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:43.555953 1150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:43.556105 1150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:43.556180 1150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:43.585414 1010 heartbeater.cc:503] Master 127.24.103.126:34275 was elected leader, sending a full tablet report...
I20260619 15:03:43.591414 1150 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:43093
I20260619 15:03:43.591430 1277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:43093 every 8 connection(s)
I20260619 15:03:43.592872 1150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:03:43.599902 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 1150
I20260619 15:03:43.600103 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:03:43.600533 24989 external_mini_cluster.cc:1428] Setting key 8ea8a18187c5bf366d87d33e84e0e39f
Jun 19 15:03:43 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881423, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:03:43.614110 1146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:43.595626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:48751 (local address 127.24.103.126:34275)
0619 15:03:43.595836 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:43.595840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:43.596979 (+ 1139us) server_negotiation.cc:407] Connection header received
0619 15:03:43.599156 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:43.599163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:43.599268 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:43.599413 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:43.601379 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:43.602505 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:43.604526 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:43.604907 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:43.605004 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:43.608444 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:43.608449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:43.608453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:43.608486 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:43.610985 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:43.611791 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:43.611795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:43.611798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:43.611927 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:43.612366 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:43.612373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:43.612381 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:43.612665 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:43.612830 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:43.613529 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 15:03:43.613775 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 15:03:43.615433 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:43.596024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:48751)
0619 15:03:43.596773 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:43.596816 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:43.598827 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:43.599672 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:43.599682 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:43.600440 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:43.601177 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:43.601204 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:43.602686 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:43.602690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:43.604393 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:43.604410 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:43.605098 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:43.605103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:43.605294 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:43.605925 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:43.605948 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:43.608229 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:03:43.611134 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:43.611141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:43.611155 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:43.611642 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:43.612040 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:03:43.612046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:03:43.612050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:03:43.612246 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:03:43.612794 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:03:43.612802 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:03:43.613164 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 15:03:43.613428 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 15:03:43.613967 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":300,"threads_started":1}
I20260619 15:03:43.616940 1278 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34275
I20260619 15:03:43.617276 1278 heartbeater.cc:464] Registering TS with master...
I20260619 15:03:43.618045 1278 heartbeater.cc:511] Master 127.24.103.126:34275 requested a full tablet report, sending...
I20260619 15:03:43.619481 821 ts_manager.cc:194] Registered new tserver with Master: d9eedaf434f241e09203060b7cada11d (127.24.103.67:43093)
I20260619 15:03:43.620265 821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:48751
I20260619 15:03:43.628890 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab.
Entry for principal oryx/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab.
WARNING: no policy specified for oryx/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab.
Entry for principal oryx/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab.
WARNING: no policy specified for oryx/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab.
Entry for principal oryx/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab.
I20260619 15:03:43.762521 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 882
Jun 19 15:03:43 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:03:43.779254 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:43.763820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.24.103.65:37319)
0619 15:03:43.764447 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:43.764454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:43.764482 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:03:43.764544 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:43.764551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:43.764784 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:43.764939 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:43.766297 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:43.767398 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:43.768677 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:43.769242 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:43.769341 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:43.773442 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:43.773456 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:43.773471 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:43.773507 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:43.776662 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:43.777317 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:43.777324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:43.777326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:43.777453 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:43.777776 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:43.777779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:43.777781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:43.778113 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:43.778332 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:43.778571 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 15:03:43.778825 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":321,"threads_started":1}
I20260619 15:03:43.780644 1005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:43.865650 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 882
I20260619 15:03:43.882891 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 1016
Jun 19 15:03:43 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:03:43.898883 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:43.883743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.24.103.66:38637)
0619 15:03:43.884284 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:43.884292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:43.884317 (+ 25us) server_negotiation.cc:407] Connection header received
0619 15:03:43.884381 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:43.884387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:43.884593 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:43.884725 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:43.886178 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:43.887316 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:43.888587 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:43.889124 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:43.889219 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:43.893432 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:43.893445 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:43.893459 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:43.893496 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:43.896177 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:43.896938 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:43.896945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:43.896947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:43.897063 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:43.897406 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:43.897408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:43.897410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:43.897737 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:43.898006 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:43.898227 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 15:03:43.898418 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":267,"threads_started":1}
I20260619 15:03:43.900156 1139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:43.983378 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 1016
I20260619 15:03:43.999557 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 1150
Jun 19 15:03:44 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:03:44.017235 1292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:44.000495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58260 (local address 127.24.103.67:43093)
0619 15:03:44.001058 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:44.001065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:44.001095 (+ 30us) server_negotiation.cc:407] Connection header received
0619 15:03:44.001154 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:44.001161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:44.001380 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:44.001557 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:44.002965 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:44.003999 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:44.005333 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:44.005929 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:44.006082 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:44.010761 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:44.010818 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:44.010838 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:44.010894 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:44.013786 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:44.014856 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:44.014865 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:44.014867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:44.015084 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:44.015529 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:44.015536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:44.015539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:44.015914 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:44.016248 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:44.016563 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 15:03:44.016858 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":265,"threads_started":1}
I20260619 15:03:44.018621 1273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:44.100659 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 1150
I20260619 15:03:44.117820 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 789
I20260619 15:03:44.119051 851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:03:44.220988 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 789
I20260619 15:03:44.241341 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34275
--webserver_interface=127.24.103.126
--webserver_port=43333
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:34275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:03:44.582026 1295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:44.582428 1295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:44.582605 1295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:44.590168 1295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:03:44.590377 1295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:44.590500 1295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:44.590623 1295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:03:44.590720 1295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:03:44.601191 1295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:34275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=43333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:44.602711 1295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:44.604183 1295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:44.617489 1303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:44.617539 1301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:44.617538 1300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:44.618436 1295 server_base.cc:1034] running on GCE node
I20260619 15:03:44.619385 1295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:44.620995 1295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:44.622289 1295 hybrid_clock.cc:648] HybridClock initialized: now 1781881424622241 us; error 56 us; skew 500 ppm
I20260619 15:03:44.622685 1295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:44 dist-test-slave-zpbw krb5kdc[776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881424, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:44.625798 1295 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:03:44.627449 1295 webserver.cc:460] Webserver started at http://127.24.103.126:43333/ using document root <none> and password file <none>
I20260619 15:03:44.628196 1295 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:44.628355 1295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:44.633881 1295 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 15:03:44.636854 1310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:44.638253 1295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:03:44.638441 1295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "f1ac2fca7cd54d94a25d87314cb2a97c"
format_stamp: "Formatted at 2026-06-19 15:03:41 on dist-test-slave-zpbw"
server_key: "e8bff2a4eddd2189f45fe63a3a83ad88"
server_key_iv: "9e13b926483c0b3d766679ab750ae339"
server_key_version: "encryptionkey@0"
I20260619 15:03:44.639964 1295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:44.657917 1295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:44.662089 1295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:44.662403 1295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:44.681558 1295 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:34275
I20260619 15:03:44.681564 1361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:34275 every 8 connection(s)
I20260619 15:03:44.683240 1295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:03:44.685736 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 1295
I20260619 15:03:44.686743 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:37319
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:34275
--webserver_port=45095
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.103.65
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:03:44.693023 1362 sys_catalog.cc:263] Verifying existing consensus state
I20260619 15:03:44.697830 1362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c: Bootstrap starting.
I20260619 15:03:44.717231 1362 log.cc:824] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c: Log is configured to *not* fsync() on all Append() calls
I20260619 15:03:44.726034 1362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:03:44.726600 1362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c: Bootstrap complete.
I20260619 15:03:44.736032 1362 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34275 } }
I20260619 15:03:44.736907 1362 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ac2fca7cd54d94a25d87314cb2a97c, State: Initialized, Role: FOLLOWER
I20260619 15:03:44.737555 1362 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34275 } }
I20260619 15:03:44.737802 1362 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:03:44.737918 1362 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:03:44.738088 1362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 1 FOLLOWER]: Advancing to term 2
I20260619 15:03:44.742168 1362 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34275 } }
I20260619 15:03:44.742736 1362 leader_election.cc:302] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1ac2fca7cd54d94a25d87314cb2a97c; no voters:
I20260619 15:03:44.743491 1362 leader_election.cc:288] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 15:03:44.743762 1366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 2 FOLLOWER]: Leader election won for term 2
I20260619 15:03:44.746222 1362 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:03:44.746295 1366 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [term 2 LEADER]: Becoming Leader. State: Replica: f1ac2fca7cd54d94a25d87314cb2a97c, State: Running, Role: LEADER
I20260619 15:03:44.747558 1366 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34275 } }
I20260619 15:03:44.752251 1368 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1ac2fca7cd54d94a25d87314cb2a97c. Latest consensus state: current_term: 2 leader_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34275 } } }
I20260619 15:03:44.752684 1368 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:44.753274 1367 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ac2fca7cd54d94a25d87314cb2a97c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34275 } } }
I20260619 15:03:44.753612 1367 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c [sys.catalog]: This master's current role is: LEADER
I20260619 15:03:44.763687 1373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:03:44.771238 1373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:03:44.772850 1373 catalog_manager.cc:1259] Loaded cluster ID: d2e9ffd417cc40cd8a3664f8770a9f93
I20260619 15:03:44.773037 1373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:03:44.777388 1373 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:03:44.779016 1373 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1ac2fca7cd54d94a25d87314cb2a97c: Loaded TSK: 0
I20260619 15:03:44.780716 1373 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 15:03:45.065129 1364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:45.065434 1364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:45.065647 1364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:45.073403 1364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:45.073534 1364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:45.073714 1364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:03:45.084273 1364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:37319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=45095
--tserver_master_addrs=127.24.103.126:34275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:45.085743 1364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:45.087147 1364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:45.101516 1390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:45.101516 1392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:45.102246 1389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:45.104450 1364 server_base.cc:1034] running on GCE node
I20260619 15:03:45.105158 1364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:45.106297 1364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:45.107529 1364 hybrid_clock.cc:648] HybridClock initialized: now 1781881425107459 us; error 71 us; skew 500 ppm
I20260619 15:03:45.107874 1364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881425, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:45.111478 1364 init.cc:377] Logged in from keytab as oryx/127.24.103.65@KRBTEST.COM (short username oryx)
I20260619 15:03:45.113652 1364 webserver.cc:460] Webserver started at http://127.24.103.65:45095/ using document root <none> and password file <none>
I20260619 15:03:45.114456 1364 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:45.114632 1364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:45.120831 1364 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 15:03:45.124155 1399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:45.125798 1364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:03:45.126012 1364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "04413a52e2af4acb9ff263187ce33842"
format_stamp: "Formatted at 2026-06-19 15:03:42 on dist-test-slave-zpbw"
server_key: "028486490233981306a7f4125fdd3b40"
server_key_iv: "d614a7849b8b3051cb72e2e9de1bba9f"
server_key_version: "encryptionkey@0"
I20260619 15:03:45.127146 1364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:45.141819 1364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:45.147327 1364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:45.147696 1364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:45.148788 1364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:45.150106 1364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:45.150192 1364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:45.150262 1364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:45.150316 1364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:45.184266 1364 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:37319
I20260619 15:03:45.184358 1511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:37319 every 8 connection(s)
I20260619 15:03:45.185807 1364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:03:45.193907 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 1364
I20260619 15:03:45.194581 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:38637
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:34275
--webserver_port=39577
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.103.66
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 15:03:45.205035 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:45.188829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:40237 (local address 127.24.103.126:34275)
0619 15:03:45.189526 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:45.189538 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:45.189838 (+ 300us) server_negotiation.cc:407] Connection header received
0619 15:03:45.192077 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:45.192091 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:45.192429 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:45.192751 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:45.194262 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.195726 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.197410 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.198098 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.198312 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:45.204010 (+ 5698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:40237: BlockingRecv error: recv got EOF from 127.24.103.65:40237 (error 108)
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":279,"threads_started":1}
I20260619 15:03:45.205009 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:45.188864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:40237)
0619 15:03:45.189664 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:45.189709 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:45.191558 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:45.192897 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:45.192905 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:45.193433 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:45.194006 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:45.194023 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.195917 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.195923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:45.197224 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:45.197238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.198353 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.198360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:45.198638 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:45.199442 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:45.199466 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:45.203603 (+ 4137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":355,"threads_started":1}
W20260619 15:03:45.205968 1512 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:34275 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.103.126:34275: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:45.216259 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:45.206596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:47373)
0619 15:03:45.206869 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:45.206891 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:45.207014 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:45.207344 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:45.207351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:45.207842 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:45.208098 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:45.208115 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.209146 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.209155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:45.210921 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:45.210941 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.211637 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.211643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:45.211832 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:45.212453 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:45.212484 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:45.216034 (+ 3550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 15:03:45.217052 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:45.206885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:47373 (local address 127.24.103.126:34275)
0619 15:03:45.207038 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:45.207042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:45.207090 (+ 48us) server_negotiation.cc:407] Connection header received
0619 15:03:45.207145 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:45.207149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:45.207207 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:45.207281 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:45.208253 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.208974 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.211075 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.211466 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.211565 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:45.216385 (+ 4820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:47373: BlockingRecv error: recv got EOF from 127.24.103.65:47373 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:45.226161 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:45.217044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:53103)
0619 15:03:45.217326 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:45.217364 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:45.217476 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:45.217988 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:45.217991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:45.218275 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:45.218422 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:45.218429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.220031 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.220034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:45.220939 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:45.220949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.221841 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.221845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:45.222014 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:45.222637 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:45.222654 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:45.225817 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171}
W20260619 15:03:45.226322 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:45.217289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:53103 (local address 127.24.103.126:34275)
0619 15:03:45.217599 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:45.217608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:45.217632 (+ 24us) server_negotiation.cc:407] Connection header received
0619 15:03:45.217698 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:45.217705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:45.217810 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:45.217969 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:45.218568 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.219867 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.221072 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.221712 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.221837 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:45.226063 (+ 4226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:53103: BlockingRecv error: recv got EOF from 127.24.103.65:53103 (error 108)
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":171}
W20260619 15:03:45.226629 1512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 15:03:45.535540 1516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:45.535835 1516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:45.535981 1516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:45.543455 1516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:45.543638 1516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:45.543812 1516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:03:45.554123 1516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:38637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=39577
--tserver_master_addrs=127.24.103.126:34275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:45.555457 1516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:45.556766 1516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:45.569626 1521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:45.569593 1524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:45.569551 1522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:45.571010 1516 server_base.cc:1034] running on GCE node
I20260619 15:03:45.571635 1516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:45.572615 1516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:45.573885 1516 hybrid_clock.cc:648] HybridClock initialized: now 1781881425573808 us; error 58 us; skew 500 ppm
I20260619 15:03:45.574249 1516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881425, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:45.577399 1516 init.cc:377] Logged in from keytab as oryx/127.24.103.66@KRBTEST.COM (short username oryx)
I20260619 15:03:45.578951 1516 webserver.cc:460] Webserver started at http://127.24.103.66:39577/ using document root <none> and password file <none>
I20260619 15:03:45.579664 1516 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:45.579813 1516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:45.585261 1516 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 15:03:45.587929 1531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:45.589325 1516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 15:03:45.589504 1516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "220bfe4aa72945cc8c1c66c2d019507a"
format_stamp: "Formatted at 2026-06-19 15:03:42 on dist-test-slave-zpbw"
server_key: "b4f5e8f152ef70841ec73d189c4aa63c"
server_key_iv: "b79dc08e5afeb5d0b94379c1cce8c4ef"
server_key_version: "encryptionkey@0"
I20260619 15:03:45.590481 1516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:45.610515 1516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:45.614713 1516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:45.615028 1516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:45.616044 1516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:45.617373 1516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:45.617496 1516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:45.617646 1516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:45.617738 1516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:45.651815 1516 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:38637
I20260619 15:03:45.651870 1643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:38637 every 8 connection(s)
I20260619 15:03:45.653280 1516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:03:45.660131 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 1516
I20260619 15:03:45.660743 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:43093
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:34275
--webserver_port=45475
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.103.67
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 15:03:45.669020 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:45.655859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:59739 (local address 127.24.103.126:34275)
0619 15:03:45.656028 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:45.656032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:45.657174 (+ 1142us) server_negotiation.cc:407] Connection header received
0619 15:03:45.659108 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:45.659114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:45.659202 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:45.659312 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:45.660945 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.661684 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.663278 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.663695 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.663826 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:45.668764 (+ 4938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:59739: BlockingRecv error: recv got EOF from 127.24.103.66:59739 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 15:03:45.669787 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:45.656333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:59739)
0619 15:03:45.656992 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:45.657037 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:45.658749 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:45.659559 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:45.659575 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:45.660114 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:45.660766 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:45.660783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.661852 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.661856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:45.663137 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:45.663148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.663930 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.663936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:45.664163 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:45.664909 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:45.664952 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:45.668542 (+ 3590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":261,"threads_started":1}
W20260619 15:03:45.670464 1644 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:34275 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.103.126:34275: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 15:03:45.680025 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:45.671209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:44681 (local address 127.24.103.126:34275)
0619 15:03:45.671421 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:45.671429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:45.671456 (+ 27us) server_negotiation.cc:407] Connection header received
0619 15:03:45.671520 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:45.671527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:45.671610 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:45.671744 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:45.672350 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.673543 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.674740 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.675592 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.675734 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:45.679845 (+ 4111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:44681: BlockingRecv error: recv got EOF from 127.24.103.66:44681 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:03:45.680030 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:45.671114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:44681)
0619 15:03:45.671306 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:45.671321 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:45.671408 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:45.671768 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:45.671771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:45.672053 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:45.672205 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:45.672212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.673704 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.673707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:45.674593 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:45.674604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.675730 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.675734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:45.675857 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:45.676465 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:45.676482 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:45.679659 (+ 3177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:45 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:45.691622 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:45.680822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:35369)
0619 15:03:45.681109 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:45.681131 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:45.681265 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:45.681625 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:45.681632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:45.682094 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:45.682329 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:45.682345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.683872 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.683881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:45.685521 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:45.685537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.686708 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.686714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:45.686977 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:45.687667 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:45.687689 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:45.691312 (+ 3623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148}
W20260619 15:03:45.691808 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:45.680887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:35369 (local address 127.24.103.126:34275)
0619 15:03:45.681083 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:45.681088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:45.681211 (+ 123us) server_negotiation.cc:407] Connection header received
0619 15:03:45.681387 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:45.681393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:45.681467 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:45.681569 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:45.682536 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.683718 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.685685 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:45.686351 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:45.686469 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:45.691582 (+ 5113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:35369: BlockingRecv error: recv got EOF from 127.24.103.66:35369 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 15:03:45.692167 1644 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 15:03:46.004070 1647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:03:46.004405 1647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:03:46.004501 1647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:03:46.011967 1647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:03:46.012108 1647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:03:46.012283 1647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:03:46.022396 1647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:43093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=45475
--tserver_master_addrs=127.24.103.126:34275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:03:46.023880 1647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:03:46.025166 1647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:03:46.038590 1652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:46.038558 1653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:03:46.038625 1655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:03:46.039429 1647 server_base.cc:1034] running on GCE node
I20260619 15:03:46.040076 1647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:03:46.041111 1647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:03:46.042341 1647 hybrid_clock.cc:648] HybridClock initialized: now 1781881426042303 us; error 41 us; skew 500 ppm
I20260619 15:03:46.042676 1647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:03:46 dist-test-slave-zpbw krb5kdc[776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881426, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:03:46.046058 1647 init.cc:377] Logged in from keytab as oryx/127.24.103.67@KRBTEST.COM (short username oryx)
I20260619 15:03:46.047761 1647 webserver.cc:460] Webserver started at http://127.24.103.67:45475/ using document root <none> and password file <none>
I20260619 15:03:46.048622 1647 fs_manager.cc:359] Metadata directory not provided
I20260619 15:03:46.048795 1647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:03:46.054355 1647 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 15:03:46.057256 1662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:46.058689 1647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 15:03:46.058931 1647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "d9eedaf434f241e09203060b7cada11d"
format_stamp: "Formatted at 2026-06-19 15:03:43 on dist-test-slave-zpbw"
server_key: "a4828babadef951c47adf914aecac9b5"
server_key_iv: "8d7c9396ed58f7eda08c80e1978b1c69"
server_key_version: "encryptionkey@0"
I20260619 15:03:46.059928 1647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:03:46.086150 1647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:03:46.090472 1647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:03:46.090705 1647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:03:46.091730 1647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:03:46.093003 1647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:03:46.093098 1647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:46.093178 1647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:03:46.093230 1647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:03:46.126758 1647 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:43093
I20260619 15:03:46.126873 1774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:43093 every 8 connection(s)
I20260619 15:03:46.128350 1647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:03:46.135921 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 1647
Jun 19 15:03:46 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:46 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 15:03:46.145780 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:46.131031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:42407 (local address 127.24.103.126:34275)
0619 15:03:46.131222 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:46.131226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:46.132381 (+ 1155us) server_negotiation.cc:407] Connection header received
0619 15:03:46.134246 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:46.134253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:46.134350 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:46.134464 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:46.136063 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.137060 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.139132 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.139850 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.139954 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:46.145475 (+ 5521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:42407: BlockingRecv error: recv got EOF from 127.24.103.67:42407 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 15:03:46.146636 1777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:46.131534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:42407)
0619 15:03:46.132220 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:46.132257 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:46.133959 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:46.134721 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:46.134729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:46.135321 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:46.135890 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:46.135906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.137300 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.137305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:46.138993 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:46.139005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.140075 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.140095 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:46.140419 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:46.141178 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:46.141201 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:46.145277 (+ 4076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":257,"threads_started":1}
W20260619 15:03:46.147399 1775 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:34275 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.103.126:34275: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
I20260619 15:03:46.152794 1779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:46.136956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.24.103.126:34275)
0619 15:03:46.137768 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:46.137775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:46.137792 (+ 17us) server_negotiation.cc:407] Connection header received
0619 15:03:46.137837 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:46.137842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:46.137905 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:46.138018 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:46.139797 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.140905 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.142191 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.142674 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.142836 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:46.145477 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:03:46.145485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:03:46.145503 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:03:46.145540 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:03:46.149164 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:46.149955 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:46.149964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:46.149972 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:46.150185 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:03:46.150567 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:03:46.150573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:03:46.150577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:03:46.151228 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:03:46.151528 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 15:03:46.152001 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 15:03:46.152509 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":601,"threads_started":1}
Jun 19 15:03:46 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:46 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:46.156195 1777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:46.148040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:44325)
0619 15:03:46.148262 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:46.148282 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:46.148397 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:46.148733 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:46.148738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:46.149077 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:46.149210 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:46.149217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.150430 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.150436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:46.151483 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:46.151495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.152242 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.152248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:46.152414 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:46.152984 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:46.153005 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:46.155904 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 15:03:46.156476 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:46.148116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:44325 (local address 127.24.103.126:34275)
0619 15:03:46.148297 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:46.148304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:46.148326 (+ 22us) server_negotiation.cc:407] Connection header received
0619 15:03:46.148503 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:46.148508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:46.148582 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:46.148704 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:46.149339 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.150301 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.151621 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.152098 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.152230 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:46.156092 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:44325: BlockingRecv error: recv got EOF from 127.24.103.67:44325 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 15:03:46 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:46 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:46.164382 1777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:46.156997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:37899)
0619 15:03:46.157156 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:46.157178 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:46.157281 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:46.157559 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:46.157561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:46.157823 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:46.157964 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:46.157971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.158967 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.158970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:46.159869 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:46.159880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.160501 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.160504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:46.160621 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:46.161237 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:46.161254 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:46.164021 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 15:03:46.164490 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:46.157098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:37899 (local address 127.24.103.126:34275)
0619 15:03:46.157309 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:46.157313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:46.157343 (+ 30us) server_negotiation.cc:407] Connection header received
0619 15:03:46.157381 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:46.157385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:46.157449 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:46.157523 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:46.158074 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.158869 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.159982 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.160385 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.160473 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:46.164232 (+ 3759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:37899: BlockingRecv error: recv got EOF from 127.24.103.67:37899 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260619 15:03:46.164812 1775 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 15:03:46 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:46 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:46.236773 1780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:46.227379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:32815)
0619 15:03:46.227900 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:46.227916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:46.228090 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:46.228613 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:46.228617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:46.229111 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:46.229293 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:46.229302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.230380 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.230385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:46.231526 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:46.231538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.232390 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.232396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:46.232573 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:46.233202 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:46.233219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:46.236435 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":275,"threads_started":1}
W20260619 15:03:46.237092 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:46.227548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:32815 (local address 127.24.103.126:34275)
0619 15:03:46.227806 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:46.227813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:46.228040 (+ 227us) server_negotiation.cc:407] Connection header received
0619 15:03:46.228394 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:46.228397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:46.228456 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:46.228592 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:46.229441 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.230232 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.231674 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.232206 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.232355 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:46.236882 (+ 4527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:32815: BlockingRecv error: recv got EOF from 127.24.103.65:32815 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Jun 19 15:03:46 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:46 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:46.701833 1781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:46.692736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:53779)
0619 15:03:46.693501 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:46.693522 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:46.693638 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:46.693926 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:46.693930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:46.694371 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:46.694737 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:46.694749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.695822 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.695826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:46.696780 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:46.696792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.697509 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.697514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:46.697649 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:46.698259 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:46.698276 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:46.701504 (+ 3228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":540,"threads_started":1}
W20260619 15:03:46.702000 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:46.692807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:53779 (local address 127.24.103.126:34275)
0619 15:03:46.693107 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:46.693117 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:46.693616 (+ 499us) server_negotiation.cc:407] Connection header received
0619 15:03:46.693739 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:46.693742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:46.693803 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:46.693882 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:46.694932 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.695699 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.696931 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:46.697360 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:46.697453 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:46.701730 (+ 4277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:53779: BlockingRecv error: recv got EOF from 127.24.103.66:53779 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 15:03:47 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:47 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:47.178089 1782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:47.165450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:39135)
0619 15:03:47.165896 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:47.165913 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:47.166031 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:47.166322 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:47.166325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:47.166760 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:47.166969 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:47.166978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:47.167948 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:47.167951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:47.169132 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:47.169145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:47.169837 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:47.169843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:47.169990 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:47.174732 (+ 4742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:47.174751 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:47.177743 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":241,"threads_started":1}
W20260619 15:03:47.178201 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:47.165505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:39135 (local address 127.24.103.126:34275)
0619 15:03:47.165746 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:47.165752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:47.165991 (+ 239us) server_negotiation.cc:407] Connection header received
0619 15:03:47.166130 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:47.166133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:47.166194 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:47.166277 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:47.167096 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:47.167825 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:47.169253 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:47.169652 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:47.169741 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:47.177960 (+ 8219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:39135: BlockingRecv error: recv got EOF from 127.24.103.67:39135 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 15:03:47 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:47 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:47.247063 1783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:47.238022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:40489)
0619 15:03:47.238527 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:47.238544 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:47.238731 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:47.239144 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:47.239149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:47.239556 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:47.239742 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:47.239752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:47.240810 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:47.240817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:47.241726 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:47.241737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:47.242526 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:47.242532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:47.242710 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:47.243483 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:47.243506 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:47.246784 (+ 3278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":279,"threads_started":1}
W20260619 15:03:47.247375 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:47.238114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:40489 (local address 127.24.103.126:34275)
0619 15:03:47.238359 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:47.238366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:47.238646 (+ 280us) server_negotiation.cc:407] Connection header received
0619 15:03:47.238917 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:47.238921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:47.238997 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:47.239119 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:47.239900 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:47.240664 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:47.241887 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:47.242359 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:47.242491 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:47.247000 (+ 4509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:40489: BlockingRecv error: recv got EOF from 127.24.103.65:40489 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 15:03:47 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:47 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:47.711652 1784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:47.703207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:38841)
0619 15:03:47.703708 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:47.703733 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:47.703888 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:47.704349 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:47.704369 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:47.704782 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:47.704975 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:47.704988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:47.706212 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:47.706216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:47.707132 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:47.707143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:47.707819 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:47.707823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:47.707929 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:47.708569 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:47.708587 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:47.711400 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":401,"threads_started":1}
W20260619 15:03:47.711831 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:47.703352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:38841 (local address 127.24.103.126:34275)
0619 15:03:47.703561 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:47.703567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:47.703804 (+ 237us) server_negotiation.cc:407] Connection header received
0619 15:03:47.704031 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:47.704042 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:47.704159 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:47.704279 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:47.705148 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:47.706089 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:47.707265 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:47.707691 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:47.707787 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:47.711621 (+ 3834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:38841: BlockingRecv error: recv got EOF from 127.24.103.66:38841 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 15:03:48 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:48 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:48.188515 1785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:48.179062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:57359)
0619 15:03:48.179614 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:48.179641 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:48.179835 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:48.180217 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:48.180222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:48.180721 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:48.180905 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:48.180914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:48.182021 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:48.182029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:48.183218 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:48.183231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:48.184143 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:48.184150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:48.184328 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:48.185015 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:48.185032 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:48.188297 (+ 3265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":279,"threads_started":1}
W20260619 15:03:48.188719 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:48.179325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:57359 (local address 127.24.103.126:34275)
0619 15:03:48.179576 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:48.179584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:48.179722 (+ 138us) server_negotiation.cc:407] Connection header received
0619 15:03:48.180001 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:48.180007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:48.180081 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:48.180193 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:48.181059 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:48.181872 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:48.183401 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:48.183962 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:48.184101 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:48.188481 (+ 4380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:57359: BlockingRecv error: recv got EOF from 127.24.103.67:57359 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 19 15:03:48 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:48 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:48.256579 1786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:48.248242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:53947)
0619 15:03:48.248757 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:48.248775 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:48.248934 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:48.249281 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:48.249286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:48.249665 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:48.249823 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:48.249831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:48.250974 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:48.250982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:48.251880 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:48.251891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:48.252576 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:48.252582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:48.252761 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:48.253333 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:48.253349 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:48.256215 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":258,"threads_started":1}
W20260619 15:03:48.256731 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:48.248362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:53947 (local address 127.24.103.126:34275)
0619 15:03:48.248619 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:48.248625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:48.248865 (+ 240us) server_negotiation.cc:407] Connection header received
0619 15:03:48.249061 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:48.249066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:48.249141 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:48.249251 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:48.249974 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:48.250838 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:48.252024 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:48.252425 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:48.252556 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:48.256449 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:53947: BlockingRecv error: recv got EOF from 127.24.103.65:53947 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 15:03:48 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:48 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:48.721608 1787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:48.712716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:49033)
0619 15:03:48.713310 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:48.713331 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:48.713527 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:48.713881 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:48.713885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:48.714270 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:48.714437 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:48.714445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:48.715680 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:48.715686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:48.716977 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:48.717014 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:48.717827 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:48.717832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:48.717953 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:48.718679 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:48.718698 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:48.721349 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":367,"threads_started":1}
W20260619 15:03:48.721763 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:48.712967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:49033 (local address 127.24.103.126:34275)
0619 15:03:48.713208 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:48.713215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:48.713449 (+ 234us) server_negotiation.cc:407] Connection header received
0619 15:03:48.713659 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:48.713664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:48.713737 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:48.713846 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:48.714583 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:48.715541 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:48.717135 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:48.717702 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:48.717795 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:48.721545 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:49033: BlockingRecv error: recv got EOF from 127.24.103.66:49033 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 15:03:49 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:49 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:49.198362 1788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:49.189633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:36261)
0619 15:03:49.190184 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:49.190201 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:49.190368 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:49.190753 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:49.190757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:49.191186 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:49.191367 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:49.191381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:49.192333 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:49.192337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:49.193295 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:49.193308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:49.194245 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:49.194252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:49.194418 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:49.195218 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:49.195243 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:49.198115 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":293,"threads_started":1}
W20260619 15:03:49.198500 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:49.189778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:36261 (local address 127.24.103.126:34275)
0619 15:03:49.190029 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:49.190036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:49.190293 (+ 257us) server_negotiation.cc:407] Connection header received
0619 15:03:49.190542 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:49.190546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:49.190616 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:49.190700 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:49.191515 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:49.192200 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:49.193474 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:49.194047 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:49.194156 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:49.198303 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:36261: BlockingRecv error: recv got EOF from 127.24.103.67:36261 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 15:03:49 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:49 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:49.265919 1789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:49.257621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:58523)
0619 15:03:49.258131 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:49.258176 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:49.258322 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:49.258717 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:49.258721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:49.259113 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:49.259263 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:49.259272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:49.260279 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:49.260282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:49.261252 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:49.261263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:49.261967 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:49.261971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:49.262106 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:49.262758 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:49.262825 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:49.265692 (+ 2867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":289,"threads_started":1}
W20260619 15:03:49.266162 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:49.257716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:58523 (local address 127.24.103.126:34275)
0619 15:03:49.257925 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:49.257934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:49.258269 (+ 335us) server_negotiation.cc:407] Connection header received
0619 15:03:49.258437 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:49.258442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:49.258539 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:49.258644 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:49.259399 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:49.260147 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:49.261388 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:49.261818 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:49.261939 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:49.265902 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:58523: BlockingRecv error: recv got EOF from 127.24.103.65:58523 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 15:03:49 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:49 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:49.731377 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:49.722614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:38877)
0619 15:03:49.723116 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:49.723135 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:49.723313 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:49.723689 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:49.723694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:49.724079 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:49.724239 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:49.724247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:49.725225 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:49.725233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:49.726211 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:49.726222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:49.727082 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:49.727097 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:49.727276 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:49.727861 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:49.727877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:49.731035 (+ 3158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":275,"threads_started":1}
W20260619 15:03:49.731551 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:49.722693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:38877 (local address 127.24.103.126:34275)
0619 15:03:49.722962 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:49.722968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:49.723228 (+ 260us) server_negotiation.cc:407] Connection header received
0619 15:03:49.723441 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:49.723447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:49.723529 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:49.723650 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:49.724384 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:49.725074 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:49.726355 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:49.726936 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:49.727069 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:49.731281 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:38877: BlockingRecv error: recv got EOF from 127.24.103.66:38877 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 15:03:50 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:50 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:50.207890 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:50.199386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:37113)
0619 15:03:50.199950 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:50.199971 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:50.200116 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:50.200490 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:50.200496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:50.200979 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:50.201177 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:50.201186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:50.202307 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:50.202311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:50.203301 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:50.203312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:50.203961 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:50.203966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:50.204081 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:50.204718 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:50.204735 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:50.207565 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":327,"threads_started":1}
W20260619 15:03:50.208024 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:50.199482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:37113 (local address 127.24.103.126:34275)
0619 15:03:50.199685 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:50.199693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:50.200069 (+ 376us) server_negotiation.cc:407] Connection header received
0619 15:03:50.200246 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:50.200251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:50.200333 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:50.200425 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:50.201331 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:50.202196 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:50.203423 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:50.203829 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:50.203939 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:50.207774 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:37113: BlockingRecv error: recv got EOF from 127.24.103.67:37113 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 15:03:50 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:50 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:50.275420 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:50.266929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:34083)
0619 15:03:50.267469 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:50.267484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:50.267656 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:50.267995 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:50.268000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:50.268455 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:50.268628 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:50.268637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:50.269675 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:50.269683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:50.270601 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:50.270612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:50.271376 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:50.271381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:50.271556 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:50.272148 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:50.272165 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:50.275046 (+ 2881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":271,"threads_started":1}
W20260619 15:03:50.275506 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:50.267007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:34083 (local address 127.24.103.126:34275)
0619 15:03:50.267297 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:50.267303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:50.267582 (+ 279us) server_negotiation.cc:407] Connection header received
0619 15:03:50.267787 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:50.267792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:50.267860 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:50.267965 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:50.268796 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:50.269530 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:50.270748 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:50.271227 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:50.271361 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:50.275276 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:34083: BlockingRecv error: recv got EOF from 127.24.103.65:34083 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Jun 19 15:03:50 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:50 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:50.740059 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:50.732412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:47235)
0619 15:03:50.732922 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:50.732942 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:50.733047 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:50.733335 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:50.733337 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:50.733715 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:50.733863 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:50.733870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:50.734888 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:50.734892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:50.735834 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:50.735845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:50.736488 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:50.736492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:50.736617 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:50.737179 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:50.737195 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:50.739820 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":264,"threads_started":1}
W20260619 15:03:50.740307 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:50.732608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:47235 (local address 127.24.103.126:34275)
0619 15:03:50.732774 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:50.732779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:50.733014 (+ 235us) server_negotiation.cc:407] Connection header received
0619 15:03:50.733143 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:50.733146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:50.733217 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:50.733295 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:50.733978 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:50.734709 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:50.735955 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:50.736370 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:50.736462 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:50.740036 (+ 3574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:47235: BlockingRecv error: recv got EOF from 127.24.103.66:47235 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 15:03:51 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:51 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:51.217114 1794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:51.208873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:48127)
0619 15:03:51.209406 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:51.209422 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:51.209565 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:51.209910 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:51.209914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:51.210438 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:51.210651 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:51.210661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:51.211756 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:51.211763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:51.212674 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:51.212685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:51.213386 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:51.213392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:51.213560 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:51.214200 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:51.214217 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:51.216853 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":288,"threads_started":1}
W20260619 15:03:51.217279 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:51.209018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:48127 (local address 127.24.103.126:34275)
0619 15:03:51.209204 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:51.209209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:51.209520 (+ 311us) server_negotiation.cc:407] Connection header received
0619 15:03:51.209688 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:51.209691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:51.209765 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:51.209876 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:51.210837 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:51.211620 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:51.212818 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:51.213245 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:51.213376 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:51.217043 (+ 3667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:48127: BlockingRecv error: recv got EOF from 127.24.103.67:48127 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 15:03:51 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:51 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:51.284881 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:51.276390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:38967)
0619 15:03:51.276901 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:51.276918 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:51.277082 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:51.277434 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:51.277438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:51.277842 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:51.277986 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:51.277994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:51.279210 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:51.279216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:51.280138 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:51.280149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:51.280865 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:51.280872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:51.281055 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:51.281628 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:51.281646 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:51.284548 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":278,"threads_started":1}
W20260619 15:03:51.285049 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:51.276460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:38967 (local address 127.24.103.126:34275)
0619 15:03:51.276729 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:51.276738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:51.277009 (+ 271us) server_negotiation.cc:407] Connection header received
0619 15:03:51.277219 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:51.277225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:51.277294 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:51.277402 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:51.278134 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:51.279076 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:51.280288 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:51.280699 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:51.280826 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:51.284749 (+ 3923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:38967: BlockingRecv error: recv got EOF from 127.24.103.65:38967 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 19 15:03:51 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:51 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:51.749946 1796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:51.741283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:36713)
0619 15:03:51.741855 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:51.741876 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:51.742041 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:51.742387 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:51.742392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:51.742885 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:51.743080 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:51.743089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:51.744218 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:51.744225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:51.745198 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:51.745209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:51.745932 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:51.745938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:51.746104 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:51.746763 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:51.746780 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:51.749620 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":311,"threads_started":1}
W20260619 15:03:51.750149 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:51.741519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:36713 (local address 127.24.103.126:34275)
0619 15:03:51.741747 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:51.741755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:51.741969 (+ 214us) server_negotiation.cc:407] Connection header received
0619 15:03:51.742182 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:51.742186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:51.742246 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:51.742356 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:51.743227 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:51.744078 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:51.745340 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:51.745774 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:51.745901 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:51.749890 (+ 3989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:36713: BlockingRecv error: recv got EOF from 127.24.103.66:36713 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 15:03:52 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:52 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:52.231685 1797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:52.218112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:51179)
0619 15:03:52.218758 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:52.218800 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:52.218928 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:52.219229 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:52.219232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:52.219687 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:52.219871 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:52.219881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:52.220892 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:52.220896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:52.221840 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:52.221852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:52.222556 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:52.222562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:52.222704 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:52.228235 (+ 5531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:52.228260 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:52.231367 (+ 3107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":386,"threads_started":1}
W20260619 15:03:52.231798 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:52.218201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:51179 (local address 127.24.103.126:34275)
0619 15:03:52.218365 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:52.218369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:52.218878 (+ 509us) server_negotiation.cc:407] Connection header received
0619 15:03:52.219034 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:52.219038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:52.219095 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:52.219173 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:52.219996 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:52.220764 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:52.221966 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:52.222401 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:52.222489 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:52.231555 (+ 9066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:51179: BlockingRecv error: recv got EOF from 127.24.103.67:51179 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 15:03:52 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:52 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:52.295584 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:52.286034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:38753)
0619 15:03:52.286674 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:52.286691 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:52.286854 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:52.287168 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:52.287171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:52.287599 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:52.287766 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:52.287775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:52.288756 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:52.288760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:52.289808 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:52.289825 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:52.290746 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:52.290752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:52.291020 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:52.291765 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:52.291784 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:52.295294 (+ 3510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":304,"threads_started":1}
W20260619 15:03:52.295753 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:52.286135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:38753 (local address 127.24.103.126:34275)
0619 15:03:52.286322 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:52.286327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:52.286822 (+ 495us) server_negotiation.cc:407] Connection header received
0619 15:03:52.286967 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:52.286972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:52.287033 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:52.287121 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:52.287899 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:52.288626 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:52.289991 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:52.290553 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:52.290661 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:52.295518 (+ 4857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:38753: BlockingRecv error: recv got EOF from 127.24.103.65:38753 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 15:03:52 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:52 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:52.759632 1799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:52.751051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:53057)
0619 15:03:52.751621 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:52.751641 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:52.751796 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:52.752225 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:52.752230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:52.752676 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:52.752842 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:52.752850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:52.753847 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:52.753854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:52.754787 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:52.754797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:52.755549 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:52.755555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:52.755729 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:52.756316 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:52.756333 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:52.759377 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":255,"threads_started":1}
W20260619 15:03:52.760033 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:52.751158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:53057 (local address 127.24.103.126:34275)
0619 15:03:52.751348 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:52.751354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:52.751732 (+ 378us) server_negotiation.cc:407] Connection header received
0619 15:03:52.752005 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:52.752009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:52.752070 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:52.752181 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:52.752982 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:52.753716 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:52.754962 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:52.755390 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:52.755519 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:52.759661 (+ 4142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:53057: BlockingRecv error: recv got EOF from 127.24.103.66:53057 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 15:03:53 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:53 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:53.240691 1800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:53.232630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:58141)
0619 15:03:53.233141 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:53.233157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:53.233268 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:53.233527 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:53.233529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:53.233905 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:53.234065 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:53.234084 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:53.235110 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:53.235124 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:53.236013 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:53.236025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:53.236752 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:53.236756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:53.236865 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:53.237404 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:53.237421 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:53.240325 (+ 2904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":214,"threads_started":1}
W20260619 15:03:53.240787 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:53.232716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:58141 (local address 127.24.103.126:34275)
0619 15:03:53.232929 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:53.232938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:53.233240 (+ 302us) server_negotiation.cc:407] Connection header received
0619 15:03:53.233371 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:53.233375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:53.233435 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:53.233514 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:53.234194 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:53.234984 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:53.236144 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:53.236619 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:53.236715 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:53.240545 (+ 3830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:58141: BlockingRecv error: recv got EOF from 127.24.103.67:58141 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 15:03:53 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:53 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:53.304586 1801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:53.296528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:60561)
0619 15:03:53.297152 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:53.297168 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:53.297289 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:53.297572 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:53.297577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:53.297964 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:53.298121 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:53.298129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:53.299121 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:53.299125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:53.300040 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:53.300050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:53.300749 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:53.300753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:53.300867 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:53.301453 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:53.301470 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:53.304250 (+ 2780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":348,"threads_started":1}
W20260619 15:03:53.304769 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:53.296685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:60561 (local address 127.24.103.126:34275)
0619 15:03:53.296915 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:53.296920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:53.297243 (+ 323us) server_negotiation.cc:407] Connection header received
0619 15:03:53.297385 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:53.297388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:53.297448 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:53.297530 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:53.298244 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:53.298988 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:53.300168 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:53.300598 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:53.300691 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:53.304472 (+ 3781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:60561: BlockingRecv error: recv got EOF from 127.24.103.65:60561 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 15:03:53 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:53 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:53.769954 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:53.760716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:44627)
0619 15:03:53.761311 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:53.761330 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:53.761497 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:53.761836 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:53.761841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:53.762307 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:53.762505 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:53.762517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:53.764227 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:53.764234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:53.765364 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:53.765376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:53.766077 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:53.766081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:53.766199 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:53.766829 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:53.766851 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:53.769726 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":279,"threads_started":1}
W20260619 15:03:53.770218 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:53.760930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:44627 (local address 127.24.103.126:34275)
0619 15:03:53.761168 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:53.761174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:53.761422 (+ 248us) server_negotiation.cc:407] Connection header received
0619 15:03:53.761628 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:53.761640 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:53.761703 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:53.761812 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:53.762803 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:53.764051 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:53.765491 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:53.765904 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:53.766001 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:53.769921 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:44627: BlockingRecv error: recv got EOF from 127.24.103.66:44627 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 15:03:54 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:54 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:54.250700 1803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:54.241698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:40355)
0619 15:03:54.242247 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:54.242263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:54.242379 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:54.242698 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:54.242704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:54.243184 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:54.243394 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:54.243403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:54.244714 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:54.244722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:54.245702 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:54.245715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:54.246478 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:54.246484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:54.246644 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:54.247401 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:54.247420 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:54.250364 (+ 2944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":301,"threads_started":1}
W20260619 15:03:54.250883 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:54.241837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:40355 (local address 127.24.103.126:34275)
0619 15:03:54.242027 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:54.242032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:54.242340 (+ 308us) server_negotiation.cc:407] Connection header received
0619 15:03:54.242479 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:54.242482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:54.242543 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:54.242640 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:54.243564 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:54.244528 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:54.245841 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:54.246292 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:54.246423 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:54.250581 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:40355: BlockingRecv error: recv got EOF from 127.24.103.67:40355 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 15:03:54 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:54 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:54.313970 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:54.305827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:55033)
0619 15:03:54.306361 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:54.306378 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:54.306489 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:54.306841 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:54.306844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:54.307228 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:54.307392 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:54.307401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:54.308368 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:54.308371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:54.309312 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:54.309322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:54.309973 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:54.309977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:54.310108 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:54.310728 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:54.310745 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:54.313588 (+ 2843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":296,"threads_started":1}
W20260619 15:03:54.313972 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:54.305889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:55033 (local address 127.24.103.126:34275)
0619 15:03:54.306069 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:54.306075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:54.306450 (+ 375us) server_negotiation.cc:407] Connection header received
0619 15:03:54.306599 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:54.306602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:54.306661 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:54.306742 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:54.307517 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:54.308253 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:54.309433 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:54.309842 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:54.309933 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:54.313784 (+ 3851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:55033: BlockingRecv error: recv got EOF from 127.24.103.65:55033 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 15:03:54 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:54 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:54.779702 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:54.771121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:43275)
0619 15:03:54.771646 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:54.771666 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:54.771837 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:54.772180 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:54.772185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:54.772598 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:54.772753 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:54.772761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:54.773802 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:54.773807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:54.774739 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:54.774750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:54.775725 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:54.775730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:54.775859 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:54.776573 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:54.776607 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:54.779452 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":279,"threads_started":1}
W20260619 15:03:54.779875 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:54.771235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:43275 (local address 127.24.103.126:34275)
0619 15:03:54.771454 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:54.771462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:54.771757 (+ 295us) server_negotiation.cc:407] Connection header received
0619 15:03:54.771964 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:54.771969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:54.772035 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:54.772154 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:54.772907 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:54.773670 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:54.774907 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:54.775551 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:54.775670 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:54.779646 (+ 3976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:43275: BlockingRecv error: recv got EOF from 127.24.103.66:43275 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 15:03:55 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:55 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:55.260356 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:55.251652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:46581)
0619 15:03:55.252198 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:55.252214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:55.252373 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:55.252751 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:55.252757 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:55.253160 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:55.253318 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:55.253327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:55.254356 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:55.254363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:55.255348 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:55.255358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:55.256115 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:55.256121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:55.256290 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:55.256877 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:55.256899 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:55.260112 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":293,"threads_started":1}
W20260619 15:03:55.260484 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:55.251785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:46581 (local address 127.24.103.126:34275)
0619 15:03:55.252018 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:55.252026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:55.252308 (+ 282us) server_negotiation.cc:407] Connection header received
0619 15:03:55.252511 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:55.252517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:55.252577 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:55.252696 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:55.253493 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:55.254218 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:55.255502 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:55.255944 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:55.256080 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:55.260287 (+ 4207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:46581: BlockingRecv error: recv got EOF from 127.24.103.67:46581 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 15:03:55 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:55 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:55.323694 1807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:55.315114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:41203)
0619 15:03:55.315671 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:55.315687 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:55.315846 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:55.316250 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:55.316253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:55.316586 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:55.316744 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:55.316751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:55.317764 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:55.317767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:55.318686 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:55.318696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:55.319507 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:55.319511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:55.319700 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:55.320466 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:55.320491 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:55.323451 (+ 2960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":307,"threads_started":1}
W20260619 15:03:55.323963 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:55.315263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:41203 (local address 127.24.103.126:34275)
0619 15:03:55.315499 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:55.315506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:55.315780 (+ 274us) server_negotiation.cc:407] Connection header received
0619 15:03:55.316002 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:55.316007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:55.316096 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:55.316178 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:55.316872 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:55.317636 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:55.318879 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:55.319356 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:55.319490 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:55.323651 (+ 4161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:41203: BlockingRecv error: recv got EOF from 127.24.103.65:41203 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 15:03:55 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:55 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:55.789270 1808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:55.780868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:46199)
0619 15:03:55.781410 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:55.781432 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:55.781573 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:55.781953 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:55.781958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:55.782453 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:55.782671 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:55.782685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:55.783798 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:55.783806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:55.784878 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:55.784889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:55.785561 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:55.785564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:55.785686 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:55.786257 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:55.786274 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:55.789044 (+ 2770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":316,"threads_started":1}
W20260619 15:03:55.789526 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:55.780976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:46199 (local address 127.24.103.126:34275)
0619 15:03:55.781191 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:55.781195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:55.781518 (+ 323us) server_negotiation.cc:407] Connection header received
0619 15:03:55.781715 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:55.781720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:55.781806 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:55.781886 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:55.782850 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:55.783641 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:55.785000 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:55.785429 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:55.785521 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:55.789265 (+ 3744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:46199: BlockingRecv error: recv got EOF from 127.24.103.66:46199 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 15:03:56 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:56 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:56.270381 1809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:56.261619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:36403)
0619 15:03:56.262155 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:56.262173 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:56.262322 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:56.262621 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:56.262624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:56.263062 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:56.263219 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:56.263228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:56.264433 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:56.264439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:56.265507 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:56.265520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:56.266371 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:56.266377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:56.266567 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:56.267179 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:56.267197 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:56.270072 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":319,"threads_started":1}
W20260619 15:03:56.270602 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:56.261888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:36403 (local address 127.24.103.126:34275)
0619 15:03:56.262136 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:56.262147 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:56.262261 (+ 114us) server_negotiation.cc:407] Connection header received
0619 15:03:56.262428 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:56.262432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:56.262492 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:56.262577 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:56.263362 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:56.264296 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:56.265660 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:56.266211 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:56.266353 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:56.270335 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:36403: BlockingRecv error: recv got EOF from 127.24.103.67:36403 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 19 15:03:56 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:56 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:56.333532 1810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:56.324732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:50439)
0619 15:03:56.325282 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:56.325299 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:56.325423 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:56.325751 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:56.325754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:56.326230 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:56.326469 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:56.326479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:56.327800 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:56.327807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:56.328797 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:56.328809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:56.329553 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:56.329559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:56.329742 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:56.330329 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:56.330346 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:56.333187 (+ 2841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":302,"threads_started":1}
W20260619 15:03:56.333690 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:56.324863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:50439 (local address 127.24.103.126:34275)
0619 15:03:56.325060 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:56.325064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:56.325382 (+ 318us) server_negotiation.cc:407] Connection header received
0619 15:03:56.325526 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:56.325530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:56.325611 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:56.325707 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:56.326627 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:56.327653 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:56.328950 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:56.329389 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:56.329530 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:56.333415 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:50439: BlockingRecv error: recv got EOF from 127.24.103.65:50439 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 15:03:56 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:56 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:56.798671 1811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:56.790266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:60127)
0619 15:03:56.790825 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:56.790845 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:56.790949 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:56.791251 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:56.791254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:56.791738 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:56.791911 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:56.791920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:56.792983 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:56.792987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:56.794015 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:56.794032 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:56.794695 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:56.794699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:56.794894 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:56.795594 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:56.795612 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:56.798370 (+ 2758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":291,"threads_started":1}
W20260619 15:03:56.798892 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:56.790365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:60127 (local address 127.24.103.126:34275)
0619 15:03:56.790548 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:56.790552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:56.790917 (+ 365us) server_negotiation.cc:407] Connection header received
0619 15:03:56.791041 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:56.791044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:56.791100 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:56.791213 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:56.792040 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:56.792850 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:56.794159 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:56.794566 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:56.794664 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:56.798631 (+ 3967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:60127: BlockingRecv error: recv got EOF from 127.24.103.66:60127 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 15:03:57 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:57 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:57.284920 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:57.271281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:38715)
0619 15:03:57.271759 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:57.271775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:57.271884 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:57.272245 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:57.272248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:57.272624 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:57.272786 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:57.272794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:57.273775 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:57.273778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:57.274630 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:57.274641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:57.275294 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:57.275298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:57.275416 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:57.281447 (+ 6031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:57.281468 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:57.284562 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":279,"threads_started":1}
W20260619 15:03:57.285050 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:57.271443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:38715 (local address 127.24.103.126:34275)
0619 15:03:57.271685 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:57.271691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:57.271848 (+ 157us) server_negotiation.cc:407] Connection header received
0619 15:03:57.272012 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:57.272020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:57.272118 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:57.272225 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:57.272936 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:57.273671 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:57.274746 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:57.275171 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:57.275263 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:57.284804 (+ 9541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:38715: BlockingRecv error: recv got EOF from 127.24.103.67:38715 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 15:03:57 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:57 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:57.342319 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:57.334457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:43345)
0619 15:03:57.334966 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:57.334982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:57.335101 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:57.335426 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:57.335428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:57.335803 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:57.335951 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:57.335959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:57.336924 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:57.336928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:57.337829 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:57.337839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:57.338540 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:57.338543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:57.338671 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:57.339306 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:57.339322 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:57.342079 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":315,"threads_started":1}
W20260619 15:03:57.342555 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:57.334502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:43345 (local address 127.24.103.126:34275)
0619 15:03:57.334724 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:57.334734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:57.335065 (+ 331us) server_negotiation.cc:407] Connection header received
0619 15:03:57.335216 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:57.335220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:57.335296 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:57.335374 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:57.336075 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:57.336809 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:57.337955 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:57.338420 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:57.338511 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:57.342293 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:43345: BlockingRecv error: recv got EOF from 127.24.103.65:43345 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 15:03:57 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:57 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:57.808777 1814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:57.799933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:48087)
0619 15:03:57.800478 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:57.800501 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:57.800635 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:57.800953 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:57.800956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:57.801347 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:57.801548 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:57.801557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:57.802946 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:57.802952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:57.804039 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:57.804052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:57.804722 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:57.804727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:57.804861 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:57.805603 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:57.805621 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:57.808539 (+ 2918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":318,"threads_started":1}
W20260619 15:03:57.809041 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:57.800039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:48087 (local address 127.24.103.126:34275)
0619 15:03:57.800225 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:57.800229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:57.800585 (+ 356us) server_negotiation.cc:407] Connection header received
0619 15:03:57.800743 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:57.800747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:57.800809 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:57.800899 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:57.801718 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:57.802710 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:57.804165 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:57.804599 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:57.804693 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:57.808776 (+ 4083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:48087: BlockingRecv error: recv got EOF from 127.24.103.66:48087 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 15:03:58 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:58 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:58.294696 1815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:58.285961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:34631)
0619 15:03:58.286498 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:58.286514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:58.286693 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:58.287066 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:58.287071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:58.287482 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:58.287633 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:58.287641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:58.288918 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:58.288924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:58.290062 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:58.290082 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:58.290843 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:58.290850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:58.291033 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:58.291683 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:58.291699 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:58.294456 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":325,"threads_started":1}
W20260619 15:03:58.295094 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:58.286053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:34631 (local address 127.24.103.126:34275)
0619 15:03:58.286333 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:58.286342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:58.286615 (+ 273us) server_negotiation.cc:407] Connection header received
0619 15:03:58.286851 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:58.286860 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:58.286923 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:58.287034 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:58.287782 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:58.288780 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:58.290219 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:58.290658 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:58.290819 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:58.294697 (+ 3878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:34631: BlockingRecv error: recv got EOF from 127.24.103.67:34631 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Jun 19 15:03:58 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:58 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:58.352021 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:58.343417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:53499)
0619 15:03:58.344004 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:58.344020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:58.344139 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:58.344494 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:58.344497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:58.344922 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:58.345074 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:58.345095 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:58.346062 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:58.346065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:58.347005 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:58.347016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:58.347734 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:58.347738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:58.347862 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:58.348483 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:58.348500 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:58.351777 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":328,"threads_started":1}
W20260619 15:03:58.352208 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:58.343521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:53499 (local address 127.24.103.126:34275)
0619 15:03:58.343706 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:58.343710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:58.344097 (+ 387us) server_negotiation.cc:407] Connection header received
0619 15:03:58.344238 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:58.344242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:58.344316 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:58.344458 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:58.345205 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:58.345948 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:58.347129 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:58.347592 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:58.347682 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:58.351968 (+ 4286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:53499: BlockingRecv error: recv got EOF from 127.24.103.65:53499 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 15:03:58 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:58 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:58.818116 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:58.809751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:46687)
0619 15:03:58.810297 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:58.810315 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:58.810465 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:58.810845 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:58.810850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:58.811245 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:58.811399 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:58.811407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:58.812483 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:58.812492 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:58.813456 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:58.813467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:58.814245 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:58.814251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:58.814417 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:58.815019 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:58.815037 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:58.817870 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":273,"threads_started":1}
W20260619 15:03:58.818383 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:58.809949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:46687 (local address 127.24.103.126:34275)
0619 15:03:58.810174 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:58.810180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:58.810390 (+ 210us) server_negotiation.cc:407] Connection header received
0619 15:03:58.810594 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:58.810599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:58.810687 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:58.810821 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:58.811539 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:58.812335 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:58.813597 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:58.814049 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:58.814187 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:58.818097 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:46687: BlockingRecv error: recv got EOF from 127.24.103.66:46687 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 15:03:59 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:59 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:59.303995 1818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:59.295651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:39173)
0619 15:03:59.296361 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:59.296384 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:59.296522 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:59.296860 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:59.296864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:59.297260 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:59.297412 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:59.297420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:59.298513 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:59.298518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:59.299510 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:59.299522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:59.300202 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:59.300206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:59.300318 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:59.300877 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:59.300893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:59.303754 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":437,"threads_started":1}
W20260619 15:03:59.304239 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:59.295702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:39173 (local address 127.24.103.126:34275)
0619 15:03:59.295935 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:59.295944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:59.296475 (+ 531us) server_negotiation.cc:407] Connection header received
0619 15:03:59.296650 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:59.296654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:59.296715 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:59.296796 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:59.297552 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:59.298334 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:59.299638 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:59.300041 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:59.300141 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:59.303971 (+ 3830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:39173: BlockingRecv error: recv got EOF from 127.24.103.67:39173 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 15:03:59 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:59 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:59.361722 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:59.353126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:48201)
0619 15:03:59.353586 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:59.353604 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:59.353728 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:59.354042 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:59.354046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:59.354413 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:59.354593 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:59.354602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:59.355702 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:59.355706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:59.356859 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:59.356881 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:59.357862 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:59.357869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:59.358065 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:59.358659 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:59.358676 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:59.361450 (+ 2774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":260,"threads_started":1}
W20260619 15:03:59.361955 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:59.353195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:48201 (local address 127.24.103.126:34275)
0619 15:03:59.353558 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:59.353564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:59.353692 (+ 128us) server_negotiation.cc:407] Connection header received
0619 15:03:59.353831 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:59.353835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:59.353911 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:59.353998 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:59.354725 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:59.355567 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:59.357074 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:59.357714 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:59.357808 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:59.361686 (+ 3878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:48201: BlockingRecv error: recv got EOF from 127.24.103.65:48201 (error 108)
Metrics: {"server-negotiator.queue_time_us":232}
Jun 19 15:03:59 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:03:59 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:03:59.832002 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:03:59.819173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:50115)
0619 15:03:59.819679 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 15:03:59.819700 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:03:59.819836 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:03:59.820124 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:03:59.820127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:03:59.820499 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:03:59.820680 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:59.820689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:59.821632 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:59.821635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:59.822540 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:03:59.822551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:03:59.823293 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:03:59.823298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:03:59.823416 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:59.828800 (+ 5384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:03:59.828821 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:03:59.831743 (+ 2922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":273,"threads_started":1}
W20260619 15:03:59.832311 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:03:59.819306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:50115 (local address 127.24.103.126:34275)
0619 15:03:59.819470 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 15:03:59.819475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:03:59.819797 (+ 322us) server_negotiation.cc:407] Connection header received
0619 15:03:59.819933 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:03:59.819936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:03:59.819995 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:03:59.820081 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:03:59.820802 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:59.821483 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:59.822673 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:03:59.823164 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:03:59.823263 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:03:59.832006 (+ 8743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:50115: BlockingRecv error: recv got EOF from 127.24.103.66:50115 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 15:04:00 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:04:00 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:04:00.313410 1821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:00.305152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.67:56341)
0619 15:04:00.305697 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:00.305717 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:00.305897 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:00.306250 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:00.306255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:00.306648 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:00.306885 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:00.306895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:00.307899 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:00.307907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:00.308819 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:00.308831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:00.309555 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:00.309561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:00.309740 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:00.310409 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:04:00.310426 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:00.313156 (+ 2730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":290,"threads_started":1}
W20260619 15:04:00.313630 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:04:00.305477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:56341 (local address 127.24.103.126:34275)
0619 15:04:00.305708 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:00.305716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:00.305817 (+ 101us) server_negotiation.cc:407] Connection header received
0619 15:04:00.306024 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:00.306029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:00.306105 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:00.306218 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:00.307026 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:00.307752 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:00.308965 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:00.309401 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:00.309546 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:00.313382 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:56341: BlockingRecv error: recv got EOF from 127.24.103.67:56341 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 15:04:00 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:04:00 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:04:00.371429 1822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:00.363130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.65:53607)
0619 15:04:00.363768 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:00.363786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:00.363903 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:00.364264 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:00.364268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:00.364714 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:00.364886 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:00.364895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:00.365939 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:00.365943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:00.366995 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:00.367007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:00.367696 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:00.367700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:00.367827 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:00.368484 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:04:00.368501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:00.371138 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":378,"threads_started":1}
W20260619 15:04:00.371718 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:04:00.363166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:53607 (local address 127.24.103.126:34275)
0619 15:04:00.363364 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:00.363368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:00.363861 (+ 493us) server_negotiation.cc:407] Connection header received
0619 15:04:00.364006 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:00.364009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:00.364114 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:00.364214 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:00.365014 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:00.365795 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:00.367122 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:00.367559 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:00.367649 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:00.371424 (+ 3775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:53607: BlockingRecv error: recv got EOF from 127.24.103.65:53607 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 15:04:00 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 15:04:00 dist-test-slave-zpbw krb5kdc[776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 15:04:00.841260 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:00.833334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34275 (local address 127.24.103.66:38707)
0619 15:04:00.833921 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:00.833940 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:00.834044 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:00.834344 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:00.834347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:00.834724 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:00.834965 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:00.834976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:00.835922 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:00.835925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:00.836855 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:00.836865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:00.837536 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:00.837540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:00.837659 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:00.838225 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:04:00.838241 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:00.841010 (+ 2769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:34275: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":328,"threads_started":1}
W20260619 15:04:00.841563 1514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:04:00.833366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:38707 (local address 127.24.103.126:34275)
0619 15:04:00.833559 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:00.833565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:00.834014 (+ 449us) server_negotiation.cc:407] Connection header received
0619 15:04:00.834157 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:00.834161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:00.834221 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:00.834304 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:00.835083 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:00.835802 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:00.836975 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:00.837414 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:00.837506 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:00.841263 (+ 3757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:38707: BlockingRecv error: recv got EOF from 127.24.103.66:38707 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 15:04:01.141911 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 1364
I20260619 15:04:01.154588 1825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:01.143143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.24.103.65:37319)
0619 15:04:01.143752 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:01.143761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:01.143794 (+ 33us) server_negotiation.cc:407] Connection header received
0619 15:04:01.143905 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:01.143913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:01.144150 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:01.144364 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:01.145899 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:01.147004 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:01.148235 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:01.148816 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:01.148950 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:01.151531 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:01.151541 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:01.151556 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:01.151590 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:01.153879 (+ 2289us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.103.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 15:04:01.154243 (+ 364us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46286: Request ticket server kudu/127.24.103.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":262,"threads_started":1}
W20260619 15:04:01.154716 1825 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46286: Request ticket server kudu/127.24.103.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 15:04:01.154811 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 1364: Not authorized: Client connection negotiation failed: client connection to 127.24.103.65:37319: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.103.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 15:04:01.155081 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 1364
I20260619 15:04:01.172310 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 1516
W20260619 15:04:01.184670 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 1516: Not authorized: Client connection negotiation failed: client connection to 127.24.103.66:38637: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.103.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 15:04:01.184804 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 1516
I20260619 15:04:01.200978 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 1647
I20260619 15:04:01.213112 1827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:01.202021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.24.103.67:43093)
0619 15:04:01.202607 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:01.202615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:01.202642 (+ 27us) server_negotiation.cc:407] Connection header received
0619 15:04:01.202707 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:01.202713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:01.203056 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:01.203253 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:01.204569 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:01.205579 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:01.206854 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:01.207426 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:01.207529 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:01.210077 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:01.210087 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:01.210101 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:01.210135 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:01.212613 (+ 2478us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.103.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 15:04:01.212887 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44586: Request ticket server kudu/127.24.103.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":319,"threads_started":1}
W20260619 15:04:01.213238 1827 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44586: Request ticket server kudu/127.24.103.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 15:04:01.213290 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 1647: Not authorized: Client connection negotiation failed: client connection to 127.24.103.67:43093: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.103.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 15:04:01.213461 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 1647
I20260619 15:04:01.229445 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 1295
I20260619 15:04:01.230909 1357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:01.346683 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 1295
2026-06-19T15:04:01Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21194 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T15:04:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:04:01Z Disabled control of system clock
I20260619 15:04:01.424877 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42249
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 15:04:01.762928 1832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:01.763233 1832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:01.763314 1832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:01.770831 1832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:04:01.770951 1832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:01.770989 1832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:01.771040 1832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:04:01.771086 1832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:04:01.780676 1832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42249
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:01.781957 1832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:01.783665 1832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:01.796026 1838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:01.796134 1837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:01.796116 1840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:01.796973 1832 server_base.cc:1034] running on GCE node
I20260619 15:04:01.797732 1832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:01.799222 1832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:01.800421 1832 hybrid_clock.cc:648] HybridClock initialized: now 1781881441800363 us; error 52 us; skew 500 ppm
I20260619 15:04:01.800756 1832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:04:01.803313 1832 webserver.cc:460] Webserver started at http://127.24.103.126:39959/ using document root <none> and password file <none>
I20260619 15:04:01.803998 1832 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:01.804088 1832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:01.804412 1832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:04:01.807670 1832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "fcca71c9bb7049a9851cc85e9825ab3d"
format_stamp: "Formatted at 2026-06-19 15:04:01 on dist-test-slave-zpbw"
server_key: "7a615de79e076896b23e3ac763517c64"
server_key_iv: "ef2c460beaad59f1ae6fcfcff582eb79"
server_key_version: "encryptionkey@0"
I20260619 15:04:01.808413 1832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "fcca71c9bb7049a9851cc85e9825ab3d"
format_stamp: "Formatted at 2026-06-19 15:04:01 on dist-test-slave-zpbw"
server_key: "7a615de79e076896b23e3ac763517c64"
server_key_iv: "ef2c460beaad59f1ae6fcfcff582eb79"
server_key_version: "encryptionkey@0"
I20260619 15:04:01.813566 1832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 15:04:01.817174 1846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:01.818534 1832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 15:04:01.818728 1832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "fcca71c9bb7049a9851cc85e9825ab3d"
format_stamp: "Formatted at 2026-06-19 15:04:01 on dist-test-slave-zpbw"
server_key: "7a615de79e076896b23e3ac763517c64"
server_key_iv: "ef2c460beaad59f1ae6fcfcff582eb79"
server_key_version: "encryptionkey@0"
I20260619 15:04:01.819041 1832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:01.838925 1832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:01.839836 1832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:01.840152 1832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:01.860597 1832 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:42249
I20260619 15:04:01.860607 1897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:42249 every 8 connection(s)
I20260619 15:04:01.862082 1832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:04:01.865868 1898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:04:01.868142 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 1832
I20260619 15:04:01.868345 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:04:01.868975 24989 external_mini_cluster.cc:1428] Setting key 504b77cdb42d42bc981410ed497b564e
I20260619 15:04:01.879364 1898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d: Bootstrap starting.
I20260619 15:04:01.883726 1898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d: Neither blocks nor log segments found. Creating new log.
I20260619 15:04:01.885138 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:01.871149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.24.103.126:42249)
0619 15:04:01.872217 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:01.872232 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:01.872285 (+ 53us) server_negotiation.cc:407] Connection header received
0619 15:04:01.874636 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:01.874669 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:01.875360 (+ 691us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:01.875835 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:01.876410 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:01.878359 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:01.879850 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:01.880881 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:01.881081 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:01.882227 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:01.882236 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:01.882254 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:01.882999 (+ 745us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:01.883122 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:01.883149 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:01.883265 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:01.883467 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 15:04:01.883823 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":340,"threads_started":1}
I20260619 15:04:01.885133 1898 log.cc:824] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d: Log is configured to *not* fsync() on all Append() calls
I20260619 15:04:01.888772 1898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d: No bootstrap required, opened a new log
I20260619 15:04:01.896430 1898 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcca71c9bb7049a9851cc85e9825ab3d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42249 } }
I20260619 15:04:01.896785 1898 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:04:01.896848 1898 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcca71c9bb7049a9851cc85e9825ab3d, State: Initialized, Role: FOLLOWER
I20260619 15:04:01.897543 1898 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcca71c9bb7049a9851cc85e9825ab3d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42249 } }
I20260619 15:04:01.897780 1898 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:04:01.897910 1898 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:04:01.898097 1898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:04:01.900728 1898 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcca71c9bb7049a9851cc85e9825ab3d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42249 } }
I20260619 15:04:01.901245 1898 leader_election.cc:302] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcca71c9bb7049a9851cc85e9825ab3d; no voters:
I20260619 15:04:01.901842 1898 leader_election.cc:288] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:04:01.901996 1903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:04:01.902667 1903 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [term 1 LEADER]: Becoming Leader. State: Replica: fcca71c9bb7049a9851cc85e9825ab3d, State: Running, Role: LEADER
I20260619 15:04:01.903239 1903 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcca71c9bb7049a9851cc85e9825ab3d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42249 } }
I20260619 15:04:01.903965 1898 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:04:01.906141 1904 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcca71c9bb7049a9851cc85e9825ab3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcca71c9bb7049a9851cc85e9825ab3d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42249 } } }
I20260619 15:04:01.906312 1905 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcca71c9bb7049a9851cc85e9825ab3d. Latest consensus state: current_term: 1 leader_uuid: "fcca71c9bb7049a9851cc85e9825ab3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcca71c9bb7049a9851cc85e9825ab3d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42249 } } }
I20260619 15:04:01.906538 1904 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [sys.catalog]: This master's current role is: LEADER
I20260619 15:04:01.906596 1905 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d [sys.catalog]: This master's current role is: LEADER
I20260619 15:04:01.908133 1911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:04:01.914053 1911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:04:01.923731 1911 catalog_manager.cc:1347] Generated new cluster ID: 80f529819432433181715b0f3c29d944
I20260619 15:04:01.923861 1911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:04:01.933338 1911 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:04:01.935213 1911 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:04:01.950464 1911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcca71c9bb7049a9851cc85e9825ab3d: Generated new TSK 0
I20260619 15:04:01.951639 1911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:04:01.964543 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42249
--builtin_ntp_servers=127.24.103.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 15:04:02.295310 1922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:02.295599 1922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:02.295683 1922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:02.302789 1922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:02.302911 1922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:02.303011 1922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:04:02.312698 1922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:02.313966 1922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:02.315779 1922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:02.329095 1927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:02.329128 1928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:02.329044 1930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:02.330049 1922 server_base.cc:1034] running on GCE node
I20260619 15:04:02.330675 1922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:02.331677 1922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:02.332902 1922 hybrid_clock.cc:648] HybridClock initialized: now 1781881442332838 us; error 61 us; skew 500 ppm
I20260619 15:04:02.333245 1922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:04:02.343216 1922 webserver.cc:460] Webserver started at http://127.24.103.65:37857/ using document root <none> and password file <none>
I20260619 15:04:02.344004 1922 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:02.344152 1922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:02.344483 1922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:04:02.347460 1922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "b630aac87ec44961b0bfb1ce9acaa8ec"
format_stamp: "Formatted at 2026-06-19 15:04:02 on dist-test-slave-zpbw"
server_key: "789edf66caa73505af260206a0bcfe4a"
server_key_iv: "d1eeb265be3d0ab5420f78e64eabb2c3"
server_key_version: "encryptionkey@0"
I20260619 15:04:02.348259 1922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "b630aac87ec44961b0bfb1ce9acaa8ec"
format_stamp: "Formatted at 2026-06-19 15:04:02 on dist-test-slave-zpbw"
server_key: "789edf66caa73505af260206a0bcfe4a"
server_key_iv: "d1eeb265be3d0ab5420f78e64eabb2c3"
server_key_version: "encryptionkey@0"
I20260619 15:04:02.353312 1922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:04:02.356869 1936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:02.358171 1922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:04:02.358361 1922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "b630aac87ec44961b0bfb1ce9acaa8ec"
format_stamp: "Formatted at 2026-06-19 15:04:02 on dist-test-slave-zpbw"
server_key: "789edf66caa73505af260206a0bcfe4a"
server_key_iv: "d1eeb265be3d0ab5420f78e64eabb2c3"
server_key_version: "encryptionkey@0"
I20260619 15:04:02.358632 1922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:02.375738 1922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:02.376626 1922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:02.376937 1922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:02.377867 1922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:04:02.379345 1922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:04:02.379475 1922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:02.379606 1922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:04:02.379693 1922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:02.413309 1922 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:42819
I20260619 15:04:02.413394 2048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:42819 every 8 connection(s)
I20260619 15:04:02.414634 1922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:04:02.417117 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 1922
I20260619 15:04:02.417313 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:04:02.417950 24989 external_mini_cluster.cc:1428] Setting key 52b4f54ce08d1f2f850c282c8a96d460
I20260619 15:04:02.421491 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42249
--builtin_ntp_servers=127.24.103.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 15:04:02.433203 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:02.417245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:58391 (local address 127.24.103.126:42249)
0619 15:04:02.424608 (+ 7363us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:02.424613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:02.424630 (+ 17us) server_negotiation.cc:407] Connection header received
0619 15:04:02.424679 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:02.424682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:02.424742 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:02.424838 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:02.426422 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:02.427305 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:02.429876 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:02.430564 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:02.430736 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:02.432199 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:02.432208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:02.432216 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:02.432244 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:02.432299 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:02.432317 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:02.432410 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:02.432629 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 15:04:02.432739 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7253,"thread_start_us":7111,"threads_started":1}
I20260619 15:04:02.434091 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:02.417952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42249 (local address 127.24.103.65:58391)
0619 15:04:02.419163 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:02.419216 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:02.420975 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:02.425178 (+ 4203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:02.425194 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:02.425325 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:02.426221 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:02.426249 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:02.427470 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:02.427478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:02.429683 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:02.429708 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:02.430887 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:02.430901 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:02.431103 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:02.431859 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:04:02.431861 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:02.431915 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:04:02.432000 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:04:02.432421 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:04:02.432431 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:04:02.432438 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 15:04:02.432591 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 15:04:02.432951 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":340,"threads_started":1}
I20260619 15:04:02.435878 2049 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42249
I20260619 15:04:02.436224 2049 heartbeater.cc:464] Registering TS with master...
I20260619 15:04:02.437001 2049 heartbeater.cc:511] Master 127.24.103.126:42249 requested a full tablet report, sending...
I20260619 15:04:02.439213 1863 ts_manager.cc:194] Registered new tserver with Master: b630aac87ec44961b0bfb1ce9acaa8ec (127.24.103.65:42819)
I20260619 15:04:02.440809 1863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.65:58391
W20260619 15:04:02.755033 2053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:02.755357 2053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:02.755537 2053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:02.763223 2053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:02.763338 2053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:02.763562 2053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:04:02.773175 2053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:02.774535 2053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:02.776393 2053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:02.789790 2061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:02.789740 2059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:02.789740 2058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:02.790856 2053 server_base.cc:1034] running on GCE node
I20260619 15:04:02.791491 2053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:02.792599 2053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:02.793834 2053 hybrid_clock.cc:648] HybridClock initialized: now 1781881442793795 us; error 56 us; skew 500 ppm
I20260619 15:04:02.794170 2053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:04:02.796556 2053 webserver.cc:460] Webserver started at http://127.24.103.66:36653/ using document root <none> and password file <none>
I20260619 15:04:02.797277 2053 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:02.797406 2053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:02.797753 2053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:04:02.800649 2053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "4f292fb5def2442d8c66028e3169f824"
format_stamp: "Formatted at 2026-06-19 15:04:02 on dist-test-slave-zpbw"
server_key: "be50600772b979a48d1de817f0a2d017"
server_key_iv: "4f205a89ed291a03878901a166ab39d6"
server_key_version: "encryptionkey@0"
I20260619 15:04:02.801476 2053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "4f292fb5def2442d8c66028e3169f824"
format_stamp: "Formatted at 2026-06-19 15:04:02 on dist-test-slave-zpbw"
server_key: "be50600772b979a48d1de817f0a2d017"
server_key_iv: "4f205a89ed291a03878901a166ab39d6"
server_key_version: "encryptionkey@0"
I20260619 15:04:02.807005 2053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 15:04:02.810571 2067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:02.811993 2053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:04:02.812170 2053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "4f292fb5def2442d8c66028e3169f824"
format_stamp: "Formatted at 2026-06-19 15:04:02 on dist-test-slave-zpbw"
server_key: "be50600772b979a48d1de817f0a2d017"
server_key_iv: "4f205a89ed291a03878901a166ab39d6"
server_key_version: "encryptionkey@0"
I20260619 15:04:02.812383 2053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:02.833869 2053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:02.834751 2053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:02.835094 2053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:02.836069 2053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:04:02.837488 2053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:04:02.837643 2053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:02.837765 2053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:04:02.837850 2053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:02.872622 2053 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:35787
I20260619 15:04:02.872716 2179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:35787 every 8 connection(s)
I20260619 15:04:02.873939 2053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:04:02.879240 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 2053
I20260619 15:04:02.879420 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:04:02.880048 24989 external_mini_cluster.cc:1428] Setting key 947a4a2d5893538ea737c23dda88fa3d
I20260619 15:04:02.883862 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42249
--builtin_ntp_servers=127.24.103.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 15:04:02.890209 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:02.876666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:48527 (local address 127.24.103.126:42249)
0619 15:04:02.876869 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:02.876875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:02.878309 (+ 1434us) server_negotiation.cc:407] Connection header received
0619 15:04:02.880821 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:02.880830 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:02.880930 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:02.881057 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:02.882892 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:02.883991 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:02.886613 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:02.887320 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:02.887431 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:02.889301 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:02.889307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:02.889312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:02.889337 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:02.889382 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:02.889403 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:02.889503 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:02.889795 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 15:04:02.889911 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:04:02.891680 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:02.877151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42249 (local address 127.24.103.66:48527)
0619 15:04:02.878136 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:02.878183 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:02.880442 (+ 2259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:02.881547 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:02.881563 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:02.881709 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:02.882622 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:02.882649 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:02.884202 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:02.884210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:02.886447 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:02.886464 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:02.887748 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:02.887756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:02.888110 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:02.888902 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:04:02.888907 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:02.888963 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:04:02.889078 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:04:02.889534 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:04:02.889546 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:04:02.889557 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 15:04:02.889748 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 15:04:02.890209 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":277,"threads_started":1}
I20260619 15:04:02.893258 2180 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42249
I20260619 15:04:02.893567 2180 heartbeater.cc:464] Registering TS with master...
I20260619 15:04:02.894312 2180 heartbeater.cc:511] Master 127.24.103.126:42249 requested a full tablet report, sending...
I20260619 15:04:02.895825 1863 ts_manager.cc:194] Registered new tserver with Master: 4f292fb5def2442d8c66028e3169f824 (127.24.103.66:35787)
I20260619 15:04:02.897049 1863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.66:48527
W20260619 15:04:03.214800 2183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:03.215096 2183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:03.215197 2183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:03.222754 2183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:03.222888 2183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:03.222994 2183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:04:03.232317 2183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:03.233590 2183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:03.235455 2183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:03.248684 2188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:03.248641 2189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:03.248713 2191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:03.249689 2183 server_base.cc:1034] running on GCE node
I20260619 15:04:03.250299 2183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:03.251372 2183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:03.252602 2183 hybrid_clock.cc:648] HybridClock initialized: now 1781881443252547 us; error 46 us; skew 500 ppm
I20260619 15:04:03.252924 2183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:04:03.255187 2183 webserver.cc:460] Webserver started at http://127.24.103.67:41021/ using document root <none> and password file <none>
I20260619 15:04:03.255884 2183 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:03.256011 2183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:03.256332 2183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:04:03.259280 2183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "380a4b6efefa44748a3c008423380f7e"
format_stamp: "Formatted at 2026-06-19 15:04:03 on dist-test-slave-zpbw"
server_key: "7808001cc6bda5c7a264eb06b51cadd6"
server_key_iv: "2282cf6cfb2918bf17c41993d599df88"
server_key_version: "encryptionkey@0"
I20260619 15:04:03.260079 2183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "380a4b6efefa44748a3c008423380f7e"
format_stamp: "Formatted at 2026-06-19 15:04:03 on dist-test-slave-zpbw"
server_key: "7808001cc6bda5c7a264eb06b51cadd6"
server_key_iv: "2282cf6cfb2918bf17c41993d599df88"
server_key_version: "encryptionkey@0"
I20260619 15:04:03.265506 2183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:04:03.268941 2197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:03.270292 2183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:04:03.270469 2183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "380a4b6efefa44748a3c008423380f7e"
format_stamp: "Formatted at 2026-06-19 15:04:03 on dist-test-slave-zpbw"
server_key: "7808001cc6bda5c7a264eb06b51cadd6"
server_key_iv: "2282cf6cfb2918bf17c41993d599df88"
server_key_version: "encryptionkey@0"
I20260619 15:04:03.270696 2183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:03.302742 2183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:03.303745 2183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:03.304044 2183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:03.305078 2183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:04:03.306515 2183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:04:03.306612 2183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:03.306694 2183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:04:03.306748 2183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:03.340373 2183 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:41357
I20260619 15:04:03.340466 2309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:41357 every 8 connection(s)
I20260619 15:04:03.341720 2183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:04:03.350401 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 2183
I20260619 15:04:03.350584 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:04:03.351109 24989 external_mini_cluster.cc:1428] Setting key 52222a36ec978fed884ec12c9f3687fc
I20260619 15:04:03.354604 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:03.344255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:37595 (local address 127.24.103.126:42249)
0619 15:04:03.344487 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:03.344493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:03.345898 (+ 1405us) server_negotiation.cc:407] Connection header received
0619 15:04:03.347685 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:03.347690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:03.347758 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:03.347846 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:03.348875 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:03.349662 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:03.351451 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:03.352005 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:03.352137 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:03.353771 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:03.353775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:03.353778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:03.353795 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:03.353826 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:03.353837 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:03.353904 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:03.354185 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 15:04:03.354260 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 15:04:03.355618 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:03.344865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42249 (local address 127.24.103.67:37595)
0619 15:04:03.345736 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:03.345776 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:03.347437 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:03.348051 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:03.348059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:03.348154 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:03.348721 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:03.348738 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:03.349807 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:03.349811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:03.351298 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:03.351317 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:03.352252 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:03.352260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:03.352595 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:03.353418 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:04:03.353421 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:03.353474 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:04:03.353575 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:04:03.353934 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:04:03.353945 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:04:03.353954 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 15:04:03.354166 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 15:04:03.354516 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":270,"threads_started":1}
I20260619 15:04:03.356864 2310 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42249
I20260619 15:04:03.357127 2310 heartbeater.cc:464] Registering TS with master...
I20260619 15:04:03.357831 2310 heartbeater.cc:511] Master 127.24.103.126:42249 requested a full tablet report, sending...
I20260619 15:04:03.359175 1863 ts_manager.cc:194] Registered new tserver with Master: 380a4b6efefa44748a3c008423380f7e (127.24.103.67:41357)
I20260619 15:04:03.359942 1863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.67:37595
I20260619 15:04:03.366618 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 15:04:03.372798 2051 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:04:03.371353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.24.103.126:42249)
0619 15:04:03.371523 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:03.371527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:03.371996 (+ 469us) server_negotiation.cc:407] Connection header received
0619 15:04:03.372138 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:03.372141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:03.372213 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:03.372295 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:03.372616 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58092: BlockingRecv error: recv got EOF from 127.0.0.1:58092 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 15:04:03.373872 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 1922
I20260619 15:04:03.382377 2319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:03.374794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.24.103.65:42819)
0619 15:04:03.375393 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:03.375400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:03.375424 (+ 24us) server_negotiation.cc:407] Connection header received
0619 15:04:03.375476 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:03.375482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:03.375766 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:03.375927 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:03.376340 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:03.377375 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:03.378676 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:03.379309 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:03.379452 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:03.380584 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:03.380593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:03.380606 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:03.381410 (+ 804us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:03.381479 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:03.381511 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:03.381654 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:03.381837 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 15:04:03.382012 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":246,"threads_started":1}
I20260619 15:04:03.383680 2044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:03.464690 2049 heartbeater.cc:503] Master 127.24.103.126:42249 was elected leader, sending a full tablet report...
I20260619 15:04:03.466117 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 1922
W20260619 15:04:03.482813 1848 connection.cc:535] server connection from 127.24.103.65:58391 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 15:04:03.482992 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 2053
I20260619 15:04:03.490686 2321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:03.483927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.24.103.66:35787)
0619 15:04:03.484486 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:03.484493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:03.484521 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:04:03.484580 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:03.484586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:03.484818 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:03.484952 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:03.485306 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:03.486313 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:03.487583 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:03.488146 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:03.488243 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:03.489261 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:03.489268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:03.489282 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:03.489863 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:03.489907 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:03.489925 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:03.490010 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:03.490183 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 15:04:03.490291 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":292,"threads_started":1}
I20260619 15:04:03.492046 2175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:03.573266 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 2053
I20260619 15:04:03.589787 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 2183
I20260619 15:04:03.598043 2323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:03.590669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.24.103.67:41357)
0619 15:04:03.591413 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:03.591420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:03.591448 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:04:03.591507 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:03.591513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:03.591789 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:03.591987 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:03.592445 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:03.593452 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:03.594723 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:03.595365 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:03.595504 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:03.596618 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:03.596626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:03.596640 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:03.597213 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:03.597257 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:03.597274 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:03.597409 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:03.597571 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 15:04:03.597700 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":450,"threads_started":1}
I20260619 15:04:03.599429 2305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:03.680691 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 2183
I20260619 15:04:03.696271 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 1832
I20260619 15:04:03.697672 1893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:03.800387 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 1832
2026-06-19T15:04:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2448 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T15:04:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:04:03Z Disabled control of system clock
I20260619 15:04:03.872413 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38039
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 15:04:04.209616 2329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:04.209954 2329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:04.210039 2329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:04.217175 2329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:04:04.217290 2329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:04.217329 2329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:04.217388 2329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:04:04.217432 2329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:04:04.226581 2329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38039
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:04.227932 2329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:04.229683 2329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:04.241523 2337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:04.241565 2334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:04.241443 2335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:04.242302 2329 server_base.cc:1034] running on GCE node
I20260619 15:04:04.243161 2329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:04.244580 2329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:04.245800 2329 hybrid_clock.cc:648] HybridClock initialized: now 1781881444245769 us; error 33 us; skew 500 ppm
I20260619 15:04:04.246155 2329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:04:04.248401 2329 webserver.cc:460] Webserver started at http://127.24.103.126:44337/ using document root <none> and password file <none>
I20260619 15:04:04.249122 2329 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:04.249269 2329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:04.249604 2329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:04:04.252624 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "3511a0a8bc914a62966fe97b26660ca0"
format_stamp: "Formatted at 2026-06-19 15:04:04 on dist-test-slave-zpbw"
server_key: "5250fd70b85bf93eed2dea2b7554f424"
server_key_iv: "e7cbe73fb50c8e9b80f2b537827b9b03"
server_key_version: "encryptionkey@0"
I20260619 15:04:04.253456 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "3511a0a8bc914a62966fe97b26660ca0"
format_stamp: "Formatted at 2026-06-19 15:04:04 on dist-test-slave-zpbw"
server_key: "5250fd70b85bf93eed2dea2b7554f424"
server_key_iv: "e7cbe73fb50c8e9b80f2b537827b9b03"
server_key_version: "encryptionkey@0"
I20260619 15:04:04.258931 2329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 15:04:04.262426 2343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:04.263847 2329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:04:04.264053 2329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "3511a0a8bc914a62966fe97b26660ca0"
format_stamp: "Formatted at 2026-06-19 15:04:04 on dist-test-slave-zpbw"
server_key: "5250fd70b85bf93eed2dea2b7554f424"
server_key_iv: "e7cbe73fb50c8e9b80f2b537827b9b03"
server_key_version: "encryptionkey@0"
I20260619 15:04:04.264297 2329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:04.270104 2329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:04.270983 2329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:04.271291 2329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:04.289700 2329 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:38039
I20260619 15:04:04.289729 2394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:38039 every 8 connection(s)
I20260619 15:04:04.291242 2329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:04:04.295058 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:04:04.297216 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 2329
I20260619 15:04:04.297452 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:04:04.298256 24989 external_mini_cluster.cc:1428] Setting key 787ad75a9271d314c707c0015f7ede0e
I20260619 15:04:04.306929 2398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:04.300468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.24.103.126:38039)
0619 15:04:04.301548 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:04.301560 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:04.301608 (+ 48us) server_negotiation.cc:407] Connection header received
0619 15:04:04.303300 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:04.303324 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:04.303768 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:04.304103 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:04.305244 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:04.305253 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:04.305266 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:04.305279 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:04.305344 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:04.305359 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:04.305528 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:04.305649 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 15:04:04.305956 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":420,"threads_started":1}
I20260619 15:04:04.307826 2395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0: Bootstrap starting.
I20260619 15:04:04.310987 2395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0: Neither blocks nor log segments found. Creating new log.
I20260619 15:04:04.312045 2395 log.cc:824] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0: Log is configured to *not* fsync() on all Append() calls
I20260619 15:04:04.314733 2395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0: No bootstrap required, opened a new log
I20260619 15:04:04.321902 2395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3511a0a8bc914a62966fe97b26660ca0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38039 } }
I20260619 15:04:04.322325 2395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:04:04.322453 2395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3511a0a8bc914a62966fe97b26660ca0, State: Initialized, Role: FOLLOWER
I20260619 15:04:04.323114 2395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3511a0a8bc914a62966fe97b26660ca0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38039 } }
I20260619 15:04:04.323359 2395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:04:04.323506 2395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:04:04.323679 2395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:04:04.325839 2395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3511a0a8bc914a62966fe97b26660ca0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38039 } }
I20260619 15:04:04.326457 2395 leader_election.cc:302] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3511a0a8bc914a62966fe97b26660ca0; no voters:
I20260619 15:04:04.327153 2395 leader_election.cc:288] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:04:04.327286 2400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:04:04.328009 2400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [term 1 LEADER]: Becoming Leader. State: Replica: 3511a0a8bc914a62966fe97b26660ca0, State: Running, Role: LEADER
I20260619 15:04:04.328631 2400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3511a0a8bc914a62966fe97b26660ca0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38039 } }
I20260619 15:04:04.329389 2395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:04:04.331734 2402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3511a0a8bc914a62966fe97b26660ca0. Latest consensus state: current_term: 1 leader_uuid: "3511a0a8bc914a62966fe97b26660ca0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3511a0a8bc914a62966fe97b26660ca0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38039 } } }
I20260619 15:04:04.332165 2402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [sys.catalog]: This master's current role is: LEADER
I20260619 15:04:04.331734 2401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3511a0a8bc914a62966fe97b26660ca0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3511a0a8bc914a62966fe97b26660ca0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38039 } } }
I20260619 15:04:04.333214 2401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0 [sys.catalog]: This master's current role is: LEADER
I20260619 15:04:04.335104 2408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:04:04.340637 2408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:04:04.350603 2408 catalog_manager.cc:1347] Generated new cluster ID: 0b604b22b436462a8d5bb64173d6a54b
I20260619 15:04:04.350767 2408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:04:04.378683 2408 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:04:04.379184 2408 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:04:04.388255 2408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3511a0a8bc914a62966fe97b26660ca0: Generated new TSK 0
I20260619 15:04:04.389458 2408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:04:04.408231 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38039
--builtin_ntp_servers=127.24.103.84:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 15:04:04.748000 2419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:04.748379 2419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:04.748566 2419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:04.756239 2419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:04.756417 2419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:04.756608 2419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:04:04.766119 2419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:04.767511 2419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:04.769532 2419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:04.782912 2424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:04.782969 2425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:04.782936 2427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:04.784096 2419 server_base.cc:1034] running on GCE node
I20260619 15:04:04.784821 2419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:04.785801 2419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:04.787026 2419 hybrid_clock.cc:648] HybridClock initialized: now 1781881444786973 us; error 49 us; skew 500 ppm
I20260619 15:04:04.787351 2419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:04:04.789805 2419 webserver.cc:460] Webserver started at http://127.24.103.65:33877/ using document root <none> and password file <none>
I20260619 15:04:04.790529 2419 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:04.790663 2419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:04.791054 2419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:04:04.795866 2419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "f7e7839c971749159cfbd2b9b4397654"
format_stamp: "Formatted at 2026-06-19 15:04:04 on dist-test-slave-zpbw"
server_key: "3877d885be6e1709743214b9fc11c72a"
server_key_iv: "5944d098c5e5f3317c49efb7d2860783"
server_key_version: "encryptionkey@0"
I20260619 15:04:04.796689 2419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "f7e7839c971749159cfbd2b9b4397654"
format_stamp: "Formatted at 2026-06-19 15:04:04 on dist-test-slave-zpbw"
server_key: "3877d885be6e1709743214b9fc11c72a"
server_key_iv: "5944d098c5e5f3317c49efb7d2860783"
server_key_version: "encryptionkey@0"
I20260619 15:04:04.802019 2419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:04:04.805542 2433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:04.807130 2419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:04:04.807314 2419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "f7e7839c971749159cfbd2b9b4397654"
format_stamp: "Formatted at 2026-06-19 15:04:04 on dist-test-slave-zpbw"
server_key: "3877d885be6e1709743214b9fc11c72a"
server_key_iv: "5944d098c5e5f3317c49efb7d2860783"
server_key_version: "encryptionkey@0"
I20260619 15:04:04.807583 2419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:04.813511 2419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:04.814369 2419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:04.814610 2419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:04.815598 2419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:04:04.816970 2419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:04:04.817054 2419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:04.817137 2419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:04:04.817189 2419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:04.851615 2419 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:44621
I20260619 15:04:04.851719 2545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:44621 every 8 connection(s)
I20260619 15:04:04.853081 2419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:04:04.854974 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 2419
I20260619 15:04:04.855219 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:04:04.855949 24989 external_mini_cluster.cc:1428] Setting key 125df2af94443d235e183e93d63bed00
I20260619 15:04:04.859225 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38039
--builtin_ntp_servers=127.24.103.84:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 15:04:04.862406 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:04.856050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:49661 (local address 127.24.103.126:38039)
0619 15:04:04.856555 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:04.856562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:04.857858 (+ 1296us) server_negotiation.cc:407] Connection header received
0619 15:04:04.859884 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:04.859894 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:04.860001 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:04.860125 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:04.861493 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:04.861501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:04.861508 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:04.861532 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:04.861585 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:04.861602 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:04.861752 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:04.861916 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 15:04:04.862013 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":244,"threads_started":1}
I20260619 15:04:04.863271 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:04.856696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38039 (local address 127.24.103.65:49661)
0619 15:04:04.857706 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:04.857743 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:04.859598 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:04.860334 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:04.860343 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:04.860447 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:04.861184 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:04:04.861186 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:04.861225 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:04:04.861312 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:04:04.861716 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:04:04.861723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:04:04.861729 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 15:04:04.861979 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 15:04:04.862188 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":289,"threads_started":1}
I20260619 15:04:04.864912 2546 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38039
I20260619 15:04:04.865209 2546 heartbeater.cc:464] Registering TS with master...
I20260619 15:04:04.865819 2546 heartbeater.cc:511] Master 127.24.103.126:38039 requested a full tablet report, sending...
I20260619 15:04:04.868269 2360 ts_manager.cc:194] Registered new tserver with Master: f7e7839c971749159cfbd2b9b4397654 (127.24.103.65:44621)
W20260619 15:04:05.198499 2550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:05.198850 2550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:05.198975 2550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:05.206458 2550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:05.206573 2550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:05.206684 2550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:04:05.216470 2550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:05.217782 2550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:05.219545 2550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:05.233129 2555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:05.233187 2556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:05.233165 2558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:05.234323 2550 server_base.cc:1034] running on GCE node
I20260619 15:04:05.235054 2550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:05.236207 2550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:05.237439 2550 hybrid_clock.cc:648] HybridClock initialized: now 1781881445237405 us; error 35 us; skew 500 ppm
I20260619 15:04:05.237768 2550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:04:05.240108 2550 webserver.cc:460] Webserver started at http://127.24.103.66:36413/ using document root <none> and password file <none>
I20260619 15:04:05.240826 2550 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:05.240959 2550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:05.241292 2550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:04:05.244321 2550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "7112183310564d0f88898b1236b82bf3"
format_stamp: "Formatted at 2026-06-19 15:04:05 on dist-test-slave-zpbw"
server_key: "4da296025143422c907464c21043c1f6"
server_key_iv: "5f060a51364ab72de17864c7b8c33da7"
server_key_version: "encryptionkey@0"
I20260619 15:04:05.245173 2550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "7112183310564d0f88898b1236b82bf3"
format_stamp: "Formatted at 2026-06-19 15:04:05 on dist-test-slave-zpbw"
server_key: "4da296025143422c907464c21043c1f6"
server_key_iv: "5f060a51364ab72de17864c7b8c33da7"
server_key_version: "encryptionkey@0"
I20260619 15:04:05.250654 2550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 15:04:05.254402 2564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:05.255898 2550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:04:05.256099 2550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "7112183310564d0f88898b1236b82bf3"
format_stamp: "Formatted at 2026-06-19 15:04:05 on dist-test-slave-zpbw"
server_key: "4da296025143422c907464c21043c1f6"
server_key_iv: "5f060a51364ab72de17864c7b8c33da7"
server_key_version: "encryptionkey@0"
I20260619 15:04:05.256345 2550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:05.262233 2550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:05.263140 2550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:05.263414 2550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:05.264359 2550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:04:05.265754 2550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:04:05.265837 2550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:05.266007 2550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:04:05.266114 2550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:05.300422 2550 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:45639
I20260619 15:04:05.300453 2676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:45639 every 8 connection(s)
I20260619 15:04:05.301746 2550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:04:05.305464 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 2550
I20260619 15:04:05.305709 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:04:05.306284 24989 external_mini_cluster.cc:1428] Setting key 6788bc287b696806ba5e4ee83a69ebdc
I20260619 15:04:05.309157 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38039
--builtin_ntp_servers=127.24.103.84:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 15:04:05.310921 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:05.304456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:52255 (local address 127.24.103.126:38039)
0619 15:04:05.304676 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:05.304683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:05.306717 (+ 2034us) server_negotiation.cc:407] Connection header received
0619 15:04:05.308518 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:05.308523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:05.308586 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:05.308692 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:05.310066 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:05.310075 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:05.310092 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:05.310117 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:05.310161 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:05.310179 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:05.310307 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:05.310469 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 15:04:05.310538 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:04:05.311826 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:05.305172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38039 (local address 127.24.103.66:52255)
0619 15:04:05.306505 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:05.306558 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:05.308250 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:05.308890 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:05.308898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:05.308992 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:05.309747 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:04:05.309750 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:05.309787 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:04:05.309872 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:04:05.310279 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:04:05.310286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:04:05.310292 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 15:04:05.310501 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 15:04:05.310724 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":408,"threads_started":1}
I20260619 15:04:05.313377 2677 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38039
I20260619 15:04:05.313717 2677 heartbeater.cc:464] Registering TS with master...
I20260619 15:04:05.314297 2677 heartbeater.cc:511] Master 127.24.103.126:38039 requested a full tablet report, sending...
I20260619 15:04:05.315824 2360 ts_manager.cc:194] Registered new tserver with Master: 7112183310564d0f88898b1236b82bf3 (127.24.103.66:45639)
W20260619 15:04:05.643270 2680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:05.643581 2680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:05.643761 2680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:05.651225 2680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:05.651333 2680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:05.651531 2680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:04:05.661108 2680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:05.662532 2680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:05.664319 2680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:05.677933 2686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:05.678038 2688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:05.677959 2685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:05.678900 2680 server_base.cc:1034] running on GCE node
I20260619 15:04:05.679520 2680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:05.680563 2680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:05.681792 2680 hybrid_clock.cc:648] HybridClock initialized: now 1781881445681720 us; error 75 us; skew 500 ppm
I20260619 15:04:05.682121 2680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:04:05.684466 2680 webserver.cc:460] Webserver started at http://127.24.103.67:43637/ using document root <none> and password file <none>
I20260619 15:04:05.685204 2680 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:05.685345 2680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:05.685667 2680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:04:05.688700 2680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "d37771287a7549f5a20aedab1469e51c"
format_stamp: "Formatted at 2026-06-19 15:04:05 on dist-test-slave-zpbw"
server_key: "49810080bc4da5861b2e9333ffb0cd34"
server_key_iv: "26c637db9236db6d8f12f22f1d4e50b6"
server_key_version: "encryptionkey@0"
I20260619 15:04:05.689550 2680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "d37771287a7549f5a20aedab1469e51c"
format_stamp: "Formatted at 2026-06-19 15:04:05 on dist-test-slave-zpbw"
server_key: "49810080bc4da5861b2e9333ffb0cd34"
server_key_iv: "26c637db9236db6d8f12f22f1d4e50b6"
server_key_version: "encryptionkey@0"
I20260619 15:04:05.694968 2680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 15:04:05.698480 2694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:05.699873 2680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:04:05.700057 2680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "d37771287a7549f5a20aedab1469e51c"
format_stamp: "Formatted at 2026-06-19 15:04:05 on dist-test-slave-zpbw"
server_key: "49810080bc4da5861b2e9333ffb0cd34"
server_key_iv: "26c637db9236db6d8f12f22f1d4e50b6"
server_key_version: "encryptionkey@0"
I20260619 15:04:05.700296 2680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:05.706092 2680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:05.706969 2680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:05.707249 2680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:05.708158 2680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:04:05.709520 2680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:04:05.709647 2680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:05.709785 2680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:04:05.709874 2680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:05.742691 2680 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:45657
I20260619 15:04:05.742714 2806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:45657 every 8 connection(s)
I20260619 15:04:05.744102 2680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:04:05.751977 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:05.746645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:37463 (local address 127.24.103.126:38039)
0619 15:04:05.746866 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:05.746871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:05.748273 (+ 1402us) server_negotiation.cc:407] Connection header received
0619 15:04:05.749912 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:05.749916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:05.749973 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:05.750062 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:05.751371 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:05.751376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:05.751379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:05.751395 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:05.751425 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:05.751437 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:05.751534 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:05.751704 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 15:04:05.751754 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 15:04:05.753000 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:05.747229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38039 (local address 127.24.103.67:37463)
0619 15:04:05.748118 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:05.748154 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:05.749654 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:05.750281 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:05.750289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:05.750387 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:05.751076 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:04:05.751079 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:05.751116 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:04:05.751206 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:04:05.751522 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:04:05.751529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:04:05.751535 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 15:04:05.751753 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 15:04:05.751966 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":264,"threads_started":1}
I20260619 15:04:05.753026 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 2680
I20260619 15:04:05.753207 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:04:05.753757 24989 external_mini_cluster.cc:1428] Setting key 63ab2aaa96678fac3104b919d59ae71e
I20260619 15:04:05.754558 2807 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38039
I20260619 15:04:05.754920 2807 heartbeater.cc:464] Registering TS with master...
I20260619 15:04:05.755752 2807 heartbeater.cc:511] Master 127.24.103.126:38039 requested a full tablet report, sending...
I20260619 15:04:05.757169 2360 ts_manager.cc:194] Registered new tserver with Master: d37771287a7549f5a20aedab1469e51c (127.24.103.67:45657)
I20260619 15:04:05.769515 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 15:04:05.775764 2548 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:04:05.774014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.24.103.126:38039)
0619 15:04:05.774183 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:05.774188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:05.774666 (+ 478us) server_negotiation.cc:407] Connection header received
0619 15:04:05.774922 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:05.774927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:05.774980 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:05.775087 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:05.775474 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49812: BlockingRecv error: recv got EOF from 127.0.0.1:49812 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 15:04:05.776511 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 2419
I20260619 15:04:05.780157 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:05.777523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.24.103.65:44621)
0619 15:04:05.778062 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:05.778076 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:05.778104 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:04:05.778160 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:05.778166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:05.778379 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:05.778500 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:05.779559 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:05.779566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:05.779576 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:05.779603 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:05.779644 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:05.779659 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:05.779760 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:05.779852 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 15:04:05.779929 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":279,"threads_started":1}
I20260619 15:04:05.781332 2541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:05.862941 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 2419
I20260619 15:04:05.879245 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 2550
I20260619 15:04:05.882944 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:05.880309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.24.103.66:45639)
0619 15:04:05.880813 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:05.880821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:05.880847 (+ 26us) server_negotiation.cc:407] Connection header received
0619 15:04:05.880907 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:05.880914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:05.881131 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:05.881287 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:05.882200 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:05.882206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:05.882218 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:05.882236 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:05.882278 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:05.882292 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:05.882396 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:05.882493 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 15:04:05.882620 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":256,"threads_started":1}
I20260619 15:04:05.884213 2672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:05.965957 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 2550
I20260619 15:04:05.981617 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 2680
I20260619 15:04:05.985508 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:05.982638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.24.103.67:45657)
0619 15:04:05.983275 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:05.983282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:05.983312 (+ 30us) server_negotiation.cc:407] Connection header received
0619 15:04:05.983372 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:05.983378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:05.983598 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:05.983723 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:05.984767 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:05.984774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:05.984785 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:04:05.984800 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:05.984839 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:04:05.984852 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:05.984980 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:05.985030 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 15:04:05.985154 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":335,"threads_started":1}
I20260619 15:04:05.986765 2802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:06.068111 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 2680
I20260619 15:04:06.084164 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 2329
I20260619 15:04:06.085423 2390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:06.187202 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 2329
2026-06-19T15:04:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2387 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:04:06 dist-test-slave-zpbw krb5kdc[2825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:04:06 dist-test-slave-zpbw krb5kdc[2825](info): set up 2 sockets
Jun 19 15:04:06 dist-test-slave-zpbw krb5kdc[2825](info): commencing operation
krb5kdc: starting...
W20260619 15:04:07.311935 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:04:07 dist-test-slave-zpbw krb5kdc[2825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:04:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:04:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:04:07.546769 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:40735
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:40735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:04:07.879693 2838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:07.880066 2838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:07.880249 2838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:07.887662 2838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:04:07.887844 2838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:07.887935 2838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:07.888020 2838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:04:07.888092 2838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:04:07.898110 2838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:40735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:40735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:07.900074 2838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:07.901464 2838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:07.913506 2843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:07.913470 2846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:07.913421 2844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:07.914361 2838 server_base.cc:1034] running on GCE node
I20260619 15:04:07.915318 2838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:07.916733 2838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:07.917970 2838 hybrid_clock.cc:648] HybridClock initialized: now 1781881447917925 us; error 58 us; skew 500 ppm
I20260619 15:04:07.918365 2838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:04:07 dist-test-slave-zpbw krb5kdc[2825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881447, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:04:07.926486 2838 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:04:07.928262 2838 webserver.cc:460] Webserver started at http://127.24.103.126:39409/ using document root <none> and password file <none>
I20260619 15:04:07.928939 2838 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:07.929052 2838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:07.929311 2838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:04:07.932662 2838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "f2f0783ff738477ab244313b1b4bdd7c"
format_stamp: "Formatted at 2026-06-19 15:04:07 on dist-test-slave-zpbw"
server_key: "86f56e991253a928beff1fb612b295fd"
server_key_iv: "048c585318623807b1b84bd937a2636a"
server_key_version: "encryptionkey@0"
I20260619 15:04:07.933459 2838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "f2f0783ff738477ab244313b1b4bdd7c"
format_stamp: "Formatted at 2026-06-19 15:04:07 on dist-test-slave-zpbw"
server_key: "86f56e991253a928beff1fb612b295fd"
server_key_iv: "048c585318623807b1b84bd937a2636a"
server_key_version: "encryptionkey@0"
I20260619 15:04:07.939857 2838 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 15:04:07.944456 2853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:07.945928 2838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 15:04:07.946144 2838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "f2f0783ff738477ab244313b1b4bdd7c"
format_stamp: "Formatted at 2026-06-19 15:04:07 on dist-test-slave-zpbw"
server_key: "86f56e991253a928beff1fb612b295fd"
server_key_iv: "048c585318623807b1b84bd937a2636a"
server_key_version: "encryptionkey@0"
I20260619 15:04:07.946363 2838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:07.968390 2838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:07.973178 2838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:07.973512 2838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:07.992199 2838 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:40735
I20260619 15:04:07.992235 2904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:40735 every 8 connection(s)
I20260619 15:04:07.993619 2838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:04:07.997488 2905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:04:08.001252 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 2838
I20260619 15:04:08.001502 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:04:08.002338 24989 external_mini_cluster.cc:1428] Setting key acdf44b33879830294d5359c3898bfd7
I20260619 15:04:08.011401 2905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c: Bootstrap starting.
I20260619 15:04:08.015398 2905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c: Neither blocks nor log segments found. Creating new log.
I20260619 15:04:08.016829 2905 log.cc:824] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c: Log is configured to *not* fsync() on all Append() calls
Jun 19 15:04:08 dist-test-slave-zpbw krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:04:08.020292 2905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c: No bootstrap required, opened a new log
I20260619 15:04:08.026892 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:08.004516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.24.103.126:40735)
0619 15:04:08.005274 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:08.005286 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:08.005328 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:04:08.006990 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:08.007015 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:08.007445 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:08.007816 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:08.009733 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:08.011154 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:08.012951 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:08.013666 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:08.013797 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:08.018970 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:08.018978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:08.018993 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:08.019033 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:08.022394 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:08.023351 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:08.023358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:08.023365 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:08.023547 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:08.023967 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:08.023970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:08.023972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:08.024504 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:08.024755 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:08.025213 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 15:04:08.025668 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":355,"threads_started":1}
I20260619 15:04:08.029837 2905 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f0783ff738477ab244313b1b4bdd7c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40735 } }
I20260619 15:04:08.030351 2905 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:04:08.030484 2905 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f0783ff738477ab244313b1b4bdd7c, State: Initialized, Role: FOLLOWER
I20260619 15:04:08.031126 2905 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f0783ff738477ab244313b1b4bdd7c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40735 } }
I20260619 15:04:08.031375 2905 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:04:08.031543 2905 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:04:08.031737 2905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:04:08.034137 2905 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f0783ff738477ab244313b1b4bdd7c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40735 } }
I20260619 15:04:08.034623 2905 leader_election.cc:302] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2f0783ff738477ab244313b1b4bdd7c; no voters:
I20260619 15:04:08.035283 2905 leader_election.cc:288] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:04:08.035444 2910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:04:08.036163 2910 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [term 1 LEADER]: Becoming Leader. State: Replica: f2f0783ff738477ab244313b1b4bdd7c, State: Running, Role: LEADER
I20260619 15:04:08.036743 2910 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f0783ff738477ab244313b1b4bdd7c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40735 } }
I20260619 15:04:08.037631 2905 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:04:08.040308 2912 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2f0783ff738477ab244313b1b4bdd7c. Latest consensus state: current_term: 1 leader_uuid: "f2f0783ff738477ab244313b1b4bdd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f0783ff738477ab244313b1b4bdd7c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40735 } } }
I20260619 15:04:08.040304 2911 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2f0783ff738477ab244313b1b4bdd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f0783ff738477ab244313b1b4bdd7c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40735 } } }
I20260619 15:04:08.040789 2912 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [sys.catalog]: This master's current role is: LEADER
I20260619 15:04:08.040831 2911 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c [sys.catalog]: This master's current role is: LEADER
I20260619 15:04:08.044234 2918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:04:08.049394 2918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:04:08.058506 2918 catalog_manager.cc:1347] Generated new cluster ID: cbbab4032c7f4987a42abc2821fc8277
I20260619 15:04:08.058663 2918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:04:08.076112 2918 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:04:08.077486 2918 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:04:08.089807 2918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c: Generated new TSK 0
I20260619 15:04:08.090711 2918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:04:08.146575 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40735
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:04:08.489499 2931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:08.489806 2931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:08.489910 2931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:08.497202 2931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:08.497327 2931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:08.497442 2931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:04:08.507459 2931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:08.509008 2931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:08.510288 2931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:08.523934 2937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:08.524008 2939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:08.524005 2936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:08.525151 2931 server_base.cc:1034] running on GCE node
I20260619 15:04:08.525767 2931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:08.526875 2931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:08.528098 2931 hybrid_clock.cc:648] HybridClock initialized: now 1781881448528060 us; error 47 us; skew 500 ppm
I20260619 15:04:08.528384 2931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:04:08 dist-test-slave-zpbw krb5kdc[2825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881448, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:04:08.531741 2931 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:04:08.533252 2931 webserver.cc:460] Webserver started at http://127.24.103.65:42785/ using document root <none> and password file <none>
I20260619 15:04:08.533991 2931 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:08.534142 2931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:08.534446 2931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:04:08.538058 2931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "598d804d838548a3b82835d5e7554fc9"
format_stamp: "Formatted at 2026-06-19 15:04:08 on dist-test-slave-zpbw"
server_key: "e491d263b95e1d345d0b918a9f7efa84"
server_key_iv: "d22bcb9530e89feb2fd6edd5ece4d94b"
server_key_version: "encryptionkey@0"
I20260619 15:04:08.539021 2931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "598d804d838548a3b82835d5e7554fc9"
format_stamp: "Formatted at 2026-06-19 15:04:08 on dist-test-slave-zpbw"
server_key: "e491d263b95e1d345d0b918a9f7efa84"
server_key_iv: "d22bcb9530e89feb2fd6edd5ece4d94b"
server_key_version: "encryptionkey@0"
I20260619 15:04:08.544505 2931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 15:04:08.548193 2946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:08.549894 2931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:04:08.550092 2931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "598d804d838548a3b82835d5e7554fc9"
format_stamp: "Formatted at 2026-06-19 15:04:08 on dist-test-slave-zpbw"
server_key: "e491d263b95e1d345d0b918a9f7efa84"
server_key_iv: "d22bcb9530e89feb2fd6edd5ece4d94b"
server_key_version: "encryptionkey@0"
I20260619 15:04:08.550354 2931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:08.573901 2931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:08.578729 2931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:08.579095 2931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:08.580144 2931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:04:08.581514 2931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:04:08.581586 2931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:08.581733 2931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:04:08.581792 2931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:08.616079 2931 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:43125
I20260619 15:04:08.616137 3058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:43125 every 8 connection(s)
I20260619 15:04:08.617401 2931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:04:08.620906 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 2931
I20260619 15:04:08.621162 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:04:08.621994 24989 external_mini_cluster.cc:1428] Setting key cebbf8499374371e7721bba0b554d0ae
Jun 19 15:04:08 dist-test-slave-zpbw krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881448, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:04:08.638168 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:08.620010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:55837 (local address 127.24.103.126:40735)
0619 15:04:08.620468 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:08.620472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:08.621544 (+ 1072us) server_negotiation.cc:407] Connection header received
0619 15:04:08.623592 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:08.623599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:08.623688 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:08.623801 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:08.625307 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:08.626030 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:08.627658 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:08.628101 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:08.628247 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:08.631962 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:08.631972 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:08.631978 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:08.632027 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:08.634621 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:08.635662 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:08.635666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:08.635668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:08.635787 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:08.636290 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:08.636293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:08.636295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:08.636556 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:08.636674 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:08.637651 (+ 977us) server_negotiation.cc:299] Negotiation successful
0619 15:04:08.637853 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":244,"threads_started":1}
I20260619 15:04:08.639787 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:08.620506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:40735 (local address 127.24.103.65:55837)
0619 15:04:08.621379 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:08.621423 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:08.623318 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:08.624057 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:08.624065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:08.624560 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:08.625141 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:08.625160 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:08.626179 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:08.626184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:08.627500 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:08.627513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:08.628357 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:08.628364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:08.628595 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:08.629282 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:04:08.629305 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:08.631673 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:04:08.634804 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:04:08.634816 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:04:08.634838 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:04:08.635531 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:04:08.635896 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:04:08.635903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:04:08.635910 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:04:08.636169 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:04:08.636697 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:04:08.636708 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:04:08.637208 (+ 500us) client_negotiation.cc:769] Sending connection context
0619 15:04:08.637510 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 15:04:08.638153 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":315,"threads_started":1}
I20260619 15:04:08.641496 3059 heartbeater.cc:347] Connected to a master server at 127.24.103.126:40735
I20260619 15:04:08.641826 3059 heartbeater.cc:464] Registering TS with master...
I20260619 15:04:08.642681 3059 heartbeater.cc:511] Master 127.24.103.126:40735 requested a full tablet report, sending...
I20260619 15:04:08.644943 2870 ts_manager.cc:194] Registered new tserver with Master: 598d804d838548a3b82835d5e7554fc9 (127.24.103.65:43125)
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260619 15:04:08.646467 2870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:55837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:04:08.676015 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40735
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:04:09.020028 3065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:09.020370 3065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:09.020536 3065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:09.028177 3065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:09.028364 3065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:09.028546 3065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:04:09.038973 3065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:09.040329 3065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:09.041726 3065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:09.055505 3073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:09.055576 3071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:09.055539 3070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:09.056636 3065 server_base.cc:1034] running on GCE node
I20260619 15:04:09.057282 3065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:09.058261 3065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:09.059489 3065 hybrid_clock.cc:648] HybridClock initialized: now 1781881449059445 us; error 68 us; skew 500 ppm
I20260619 15:04:09.059842 3065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:04:09 dist-test-slave-zpbw krb5kdc[2825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881449, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:04:09.062947 3065 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:04:09.064500 3065 webserver.cc:460] Webserver started at http://127.24.103.66:42745/ using document root <none> and password file <none>
I20260619 15:04:09.065248 3065 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:09.065388 3065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:09.065706 3065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:04:09.068722 3065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "51a7f19cdf1b428a95f6cea123f0a265"
format_stamp: "Formatted at 2026-06-19 15:04:09 on dist-test-slave-zpbw"
server_key: "c74ca941520c6e60d3918a784bd2a223"
server_key_iv: "42513f6d34a81819c058f3c300200759"
server_key_version: "encryptionkey@0"
I20260619 15:04:09.069557 3065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "51a7f19cdf1b428a95f6cea123f0a265"
format_stamp: "Formatted at 2026-06-19 15:04:09 on dist-test-slave-zpbw"
server_key: "c74ca941520c6e60d3918a784bd2a223"
server_key_iv: "42513f6d34a81819c058f3c300200759"
server_key_version: "encryptionkey@0"
I20260619 15:04:09.074688 3065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:04:09.078208 3080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:09.079733 3065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 15:04:09.079918 3065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "51a7f19cdf1b428a95f6cea123f0a265"
format_stamp: "Formatted at 2026-06-19 15:04:09 on dist-test-slave-zpbw"
server_key: "c74ca941520c6e60d3918a784bd2a223"
server_key_iv: "42513f6d34a81819c058f3c300200759"
server_key_version: "encryptionkey@0"
I20260619 15:04:09.080147 3065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:09.115873 3065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:09.120194 3065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:09.120486 3065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:09.121459 3065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:04:09.122987 3065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:04:09.123083 3065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:09.123245 3065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:04:09.123314 3065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:09.156713 3065 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:35371
I20260619 15:04:09.156757 3192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:35371 every 8 connection(s)
I20260619 15:04:09.158066 3065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:04:09.160871 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 3065
I20260619 15:04:09.161110 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:04:09.161567 24989 external_mini_cluster.cc:1428] Setting key ed66836b7826444af9bba05261f88809
Jun 19 15:04:09 dist-test-slave-zpbw krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881449, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:04:09.180483 3195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.160798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:52005 (local address 127.24.103.126:40735)
0619 15:04:09.161264 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:09.161270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:09.162349 (+ 1079us) server_negotiation.cc:407] Connection header received
0619 15:04:09.164279 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:09.164287 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:09.164370 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:09.164482 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:09.165982 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.167174 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.168787 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.169567 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.169722 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.174185 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:09.174191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:09.174194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:09.174231 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:09.177045 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:09.178060 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:09.178064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:09.178066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:09.178190 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:09.178685 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:09.178688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:09.178690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:09.179018 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:09.179128 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:09.180041 (+ 913us) server_negotiation.cc:299] Negotiation successful
0619 15:04:09.180213 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":232,"threads_started":1}
I20260619 15:04:09.182155 3196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.161408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:40735 (local address 127.24.103.66:52005)
0619 15:04:09.162184 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:09.162237 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:09.163999 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:09.164692 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:09.164700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:09.165188 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:09.165803 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.165821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.167323 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.167327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.168627 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.168639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.169790 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.169795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.170000 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.170831 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:04:09.170875 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:09.173909 (+ 3034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:04:09.177212 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:04:09.177226 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:04:09.177248 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:04:09.177931 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:04:09.178301 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:04:09.178308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:04:09.178315 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:04:09.178566 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:04:09.179140 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:04:09.179152 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:04:09.179642 (+ 490us) client_negotiation.cc:769] Sending connection context
0619 15:04:09.179974 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 15:04:09.180617 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":333,"threads_started":1}
I20260619 15:04:09.183907 3193 heartbeater.cc:347] Connected to a master server at 127.24.103.126:40735
I20260619 15:04:09.184253 3193 heartbeater.cc:464] Registering TS with master...
I20260619 15:04:09.185108 3193 heartbeater.cc:511] Master 127.24.103.126:40735 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260619 15:04:09.186718 2870 ts_manager.cc:194] Registered new tserver with Master: 51a7f19cdf1b428a95f6cea123f0a265 (127.24.103.66:35371)
I20260619 15:04:09.187619 2870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:52005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:04:09.215451 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40735
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:04:09.547638 3199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:09.547994 3199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:09.548154 3199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:09.555982 3199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:09.556160 3199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:09.556335 3199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:04:09.566296 3199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:09.567766 3199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:09.569067 3199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:09.582582 3204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:09.582492 3207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:09.582556 3205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:09.583453 3199 server_base.cc:1034] running on GCE node
I20260619 15:04:09.584344 3199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:09.585399 3199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:09.586686 3199 hybrid_clock.cc:648] HybridClock initialized: now 1781881449586637 us; error 59 us; skew 500 ppm
I20260619 15:04:09.587054 3199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:04:09 dist-test-slave-zpbw krb5kdc[2825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881449, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:04:09.590134 3199 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:04:09.591667 3199 webserver.cc:460] Webserver started at http://127.24.103.67:40171/ using document root <none> and password file <none>
I20260619 15:04:09.592396 3199 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:09.592540 3199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:09.592851 3199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:04:09.595796 3199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "0325e7350d6d4ff39d1d286db2d02264"
format_stamp: "Formatted at 2026-06-19 15:04:09 on dist-test-slave-zpbw"
server_key: "615baa26452f60b886ae0ca01a2d1c99"
server_key_iv: "a620cf1a4b6d5080622d54707f0578c1"
server_key_version: "encryptionkey@0"
I20260619 15:04:09.596526 3199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "0325e7350d6d4ff39d1d286db2d02264"
format_stamp: "Formatted at 2026-06-19 15:04:09 on dist-test-slave-zpbw"
server_key: "615baa26452f60b886ae0ca01a2d1c99"
server_key_iv: "a620cf1a4b6d5080622d54707f0578c1"
server_key_version: "encryptionkey@0"
I20260619 15:04:09.601511 3199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:04:09.604853 3214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:09.606215 3199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 15:04:09.606387 3199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "0325e7350d6d4ff39d1d286db2d02264"
format_stamp: "Formatted at 2026-06-19 15:04:09 on dist-test-slave-zpbw"
server_key: "615baa26452f60b886ae0ca01a2d1c99"
server_key_iv: "a620cf1a4b6d5080622d54707f0578c1"
server_key_version: "encryptionkey@0"
I20260619 15:04:09.606577 3199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:09.619441 3199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:09.623613 3199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:09.623917 3199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:09.624905 3199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:04:09.626245 3199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:04:09.626350 3199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:09.626498 3199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:04:09.626570 3199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:09.650059 3059 heartbeater.cc:503] Master 127.24.103.126:40735 was elected leader, sending a full tablet report...
I20260619 15:04:09.660460 3199 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:43989
I20260619 15:04:09.660493 3326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:43989 every 8 connection(s)
I20260619 15:04:09.661749 3199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:04:09.668598 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 3199
I20260619 15:04:09.668812 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:04:09.669371 24989 external_mini_cluster.cc:1428] Setting key 4b71800c6f054a92ac84268a300736b3
Jun 19 15:04:09 dist-test-slave-zpbw krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881449, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:04:09.681670 3195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.664289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:35341 (local address 127.24.103.126:40735)
0619 15:04:09.664463 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:09.664468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:09.665576 (+ 1108us) server_negotiation.cc:407] Connection header received
0619 15:04:09.667457 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:09.667462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:09.667572 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:09.667685 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:09.669736 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.670530 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.672237 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.672643 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.672783 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.676237 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:09.676243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:09.676246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:09.676281 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:09.678989 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:09.679706 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:09.679709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:09.679711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:09.679830 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:09.680213 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:09.680216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:09.680218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:09.680477 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:09.680599 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:09.681256 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 15:04:09.681434 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 15:04:09.682762 3329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.664746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:40735 (local address 127.24.103.67:35341)
0619 15:04:09.665417 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:09.665453 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:09.667178 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:09.667923 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:09.667932 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:09.668664 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:09.669528 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.669555 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.670686 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.670692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.672074 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.672095 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.672871 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.672881 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.673136 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.673782 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:04:09.673805 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:09.676007 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:04:09.679128 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:04:09.679136 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:04:09.679149 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:04:09.679579 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:04:09.679925 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:04:09.679928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:04:09.679930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:04:09.680111 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:04:09.680578 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:04:09.680591 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:04:09.680926 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 15:04:09.681154 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 15:04:09.681558 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":259,"threads_started":1}
I20260619 15:04:09.684372 3327 heartbeater.cc:347] Connected to a master server at 127.24.103.126:40735
I20260619 15:04:09.684718 3327 heartbeater.cc:464] Registering TS with master...
I20260619 15:04:09.685572 3327 heartbeater.cc:511] Master 127.24.103.126:40735 requested a full tablet report, sending...
I20260619 15:04:09.687088 2869 ts_manager.cc:194] Registered new tserver with Master: 0325e7350d6d4ff39d1d286db2d02264 (127.24.103.67:43989)
I20260619 15:04:09.687980 2869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:35341
I20260619 15:04:09.697602 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:04:09.715819 3195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.702063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.24.103.126:40735)
0619 15:04:09.702238 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:09.702243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:09.702825 (+ 582us) server_negotiation.cc:407] Connection header received
0619 15:04:09.703015 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:09.703018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:09.703087 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:09.703168 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:09.704497 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.705329 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.706646 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.707197 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.707339 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.709580 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:09.709587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:09.709594 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:09.709634 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:09.712945 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:09.713652 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:09.713658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:09.713662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:09.713800 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:09.714192 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:09.714197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:09.714201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:09.714527 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:09.714830 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:09.715251 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 15:04:09.715471 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 15:04:09.720054 2869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:04:09.726958 2869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:04:09.760550 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.739110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.24.103.65:43125)
0619 15:04:09.751612 (+ 12502us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:09.751633 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:09.751679 (+ 46us) server_negotiation.cc:407] Connection header received
0619 15:04:09.751801 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:09.751811 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:09.752110 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:09.753205 (+ 1095us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:04:09.753456 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.755099 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.758099 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.759360 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.759514 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.759764 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:09.760038 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 15:04:09.760207 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12324,"thread_start_us":12157,"threads_started":1}
I20260619 15:04:09.760597 3340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.738812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:43125 (local address 127.0.0.1:44716)
0619 15:04:09.749420 (+ 10608us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:09.749463 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:09.749679 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:09.752390 (+ 2711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:09.752401 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:09.752444 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:04:09.752978 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.752993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.755262 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.755268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.757922 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.757937 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.759567 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.759573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.759783 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.759844 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 15:04:09.759986 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 15:04:09.760192 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10328,"thread_start_us":12863,"threads_started":1}
I20260619 15:04:09.761624 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.739631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.24.103.67:43989)
0619 15:04:09.752755 (+ 13124us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:09.752764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:09.752806 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:04:09.752900 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:09.752910 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:09.753156 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:09.754129 (+ 973us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:04:09.754397 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.755974 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.758808 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.760141 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.760276 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.760477 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:09.761128 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 15:04:09.761285 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12941,"thread_start_us":13652,"threads_started":1}
I20260619 15:04:09.761667 3343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.741479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:35371 (local address 127.0.0.1:50538)
0619 15:04:09.753024 (+ 11545us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:09.753049 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:09.753209 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:09.753977 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:09.753981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:09.754006 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:04:09.754435 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.754448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.756257 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.756262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.758491 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.758502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.760640 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.760648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.760934 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.760961 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 15:04:09.761028 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 15:04:09.761166 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11403,"mutex_wait_us":168,"thread_start_us":11221,"threads_started":1}
I20260619 15:04:09.762226 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.738979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:43989 (local address 127.0.0.1:42304)
0619 15:04:09.752009 (+ 13030us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:09.752029 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:09.752152 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:09.753624 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:09.753635 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:09.753660 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:04:09.754193 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.754210 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.756129 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.756136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.758632 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.758647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.760658 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.760664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.760894 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.760931 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 15:04:09.761021 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 15:04:09.761926 (+ 905us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12828,"mutex_wait_us":810,"thread_start_us":10007,"threads_started":1}
I20260619 15:04:09.763027 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.742032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.24.103.66:35371)
0619 15:04:09.752490 (+ 10458us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:09.752502 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:09.753336 (+ 834us) server_negotiation.cc:407] Connection header received
0619 15:04:09.753437 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:09.753444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:09.753645 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:09.753827 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:04:09.754652 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.755997 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.758645 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.760141 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.760276 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.760444 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:09.762577 (+ 2133us) server_negotiation.cc:299] Negotiation successful
0619 15:04:09.762745 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10310,"thread_start_us":10213,"threads_started":1}
I20260619 15:04:09.765440 2994 tablet_service.cc:1467] Processing CreateTablet for tablet 13475478b6744ece8982259ecf658097 (DEFAULT_TABLE table=test-table [id=700b9d413157495c8edfb3fed18d77d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:04:09.765718 3128 tablet_service.cc:1467] Processing CreateTablet for tablet 13475478b6744ece8982259ecf658097 (DEFAULT_TABLE table=test-table [id=700b9d413157495c8edfb3fed18d77d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:04:09.765801 3262 tablet_service.cc:1467] Processing CreateTablet for tablet 13475478b6744ece8982259ecf658097 (DEFAULT_TABLE table=test-table [id=700b9d413157495c8edfb3fed18d77d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:04:09.767043 3128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13475478b6744ece8982259ecf658097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:04:09.767072 3262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13475478b6744ece8982259ecf658097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:04:09.767132 2994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13475478b6744ece8982259ecf658097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:04:09.778610 3345 tablet_bootstrap.cc:492] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265: Bootstrap starting.
I20260619 15:04:09.782065 3346 tablet_bootstrap.cc:492] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9: Bootstrap starting.
I20260619 15:04:09.783289 3345 tablet_bootstrap.cc:654] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265: Neither blocks nor log segments found. Creating new log.
I20260619 15:04:09.784771 3345 log.cc:824] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265: Log is configured to *not* fsync() on all Append() calls
I20260619 15:04:09.785812 3346 tablet_bootstrap.cc:654] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9: Neither blocks nor log segments found. Creating new log.
I20260619 15:04:09.786295 3347 tablet_bootstrap.cc:492] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264: Bootstrap starting.
I20260619 15:04:09.787171 3346 log.cc:824] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9: Log is configured to *not* fsync() on all Append() calls
I20260619 15:04:09.788393 3345 tablet_bootstrap.cc:492] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265: No bootstrap required, opened a new log
I20260619 15:04:09.788672 3345 ts_tablet_manager.cc:1397] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 15:04:09.790112 3346 tablet_bootstrap.cc:492] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9: No bootstrap required, opened a new log
I20260619 15:04:09.790400 3346 ts_tablet_manager.cc:1397] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 15:04:09.791559 3347 tablet_bootstrap.cc:654] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264: Neither blocks nor log segments found. Creating new log.
I20260619 15:04:09.792790 3347 log.cc:824] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264: Log is configured to *not* fsync() on all Append() calls
I20260619 15:04:09.796075 3347 tablet_bootstrap.cc:492] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264: No bootstrap required, opened a new log
I20260619 15:04:09.796376 3347 ts_tablet_manager.cc:1397] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 15:04:09.798157 3346 raft_consensus.cc:348] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598d804d838548a3b82835d5e7554fc9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43125 } } peers { permanent_uuid: "0325e7350d6d4ff39d1d286db2d02264" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43989 } } peers { permanent_uuid: "51a7f19cdf1b428a95f6cea123f0a265" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35371 } }
I20260619 15:04:09.798520 3346 raft_consensus.cc:374] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:04:09.798652 3346 raft_consensus.cc:729] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598d804d838548a3b82835d5e7554fc9, State: Initialized, Role: FOLLOWER
I20260619 15:04:09.799352 3346 consensus_queue.cc:260] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598d804d838548a3b82835d5e7554fc9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43125 } } peers { permanent_uuid: "0325e7350d6d4ff39d1d286db2d02264" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43989 } } peers { permanent_uuid: "51a7f19cdf1b428a95f6cea123f0a265" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35371 } }
I20260619 15:04:09.799628 3345 raft_consensus.cc:348] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598d804d838548a3b82835d5e7554fc9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43125 } } peers { permanent_uuid: "0325e7350d6d4ff39d1d286db2d02264" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43989 } } peers { permanent_uuid: "51a7f19cdf1b428a95f6cea123f0a265" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35371 } }
I20260619 15:04:09.800053 3345 raft_consensus.cc:374] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:04:09.800143 3345 raft_consensus.cc:729] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a7f19cdf1b428a95f6cea123f0a265, State: Initialized, Role: FOLLOWER
I20260619 15:04:09.800796 3345 consensus_queue.cc:260] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598d804d838548a3b82835d5e7554fc9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43125 } } peers { permanent_uuid: "0325e7350d6d4ff39d1d286db2d02264" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43989 } } peers { permanent_uuid: "51a7f19cdf1b428a95f6cea123f0a265" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35371 } }
I20260619 15:04:09.804298 3346 ts_tablet_manager.cc:1428] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260619 15:04:09.805408 3193 heartbeater.cc:503] Master 127.24.103.126:40735 was elected leader, sending a full tablet report...
I20260619 15:04:09.806059 3345 ts_tablet_manager.cc:1428] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20260619 15:04:09.808118 3347 raft_consensus.cc:348] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598d804d838548a3b82835d5e7554fc9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43125 } } peers { permanent_uuid: "0325e7350d6d4ff39d1d286db2d02264" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43989 } } peers { permanent_uuid: "51a7f19cdf1b428a95f6cea123f0a265" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35371 } }
I20260619 15:04:09.808598 3347 raft_consensus.cc:374] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:04:09.808702 3347 raft_consensus.cc:729] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0325e7350d6d4ff39d1d286db2d02264, State: Initialized, Role: FOLLOWER
I20260619 15:04:09.809377 3347 consensus_queue.cc:260] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598d804d838548a3b82835d5e7554fc9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43125 } } peers { permanent_uuid: "0325e7350d6d4ff39d1d286db2d02264" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43989 } } peers { permanent_uuid: "51a7f19cdf1b428a95f6cea123f0a265" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35371 } }
I20260619 15:04:09.810551 3327 heartbeater.cc:503] Master 127.24.103.126:40735 was elected leader, sending a full tablet report...
I20260619 15:04:09.811267 3347 ts_tablet_manager.cc:1428] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260619 15:04:09.869845 3060 tablet.cc:2369] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:04:09.901351 3352 raft_consensus.cc:482] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:04:09.901701 3352 raft_consensus.cc:504] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598d804d838548a3b82835d5e7554fc9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43125 } } peers { permanent_uuid: "0325e7350d6d4ff39d1d286db2d02264" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43989 } } peers { permanent_uuid: "51a7f19cdf1b428a95f6cea123f0a265" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35371 } }
I20260619 15:04:09.903520 3352 leader_election.cc:288] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598d804d838548a3b82835d5e7554fc9 (127.24.103.65:43125), 0325e7350d6d4ff39d1d286db2d02264 (127.24.103.67:43989)
W20260619 15:04:09.910064 3194 tablet.cc:2369] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:04:09.912879 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.904172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:58567 (local address 127.24.103.65:43125)
0619 15:04:09.904372 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:09.904380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:09.904445 (+ 65us) server_negotiation.cc:407] Connection header received
0619 15:04:09.904731 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:09.904738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:09.904813 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:09.904873 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:04:09.905510 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.906709 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.910195 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.911481 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.911672 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.912376 (+ 704us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:09.912441 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 15:04:09.912584 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 15:04:09.912971 3354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.903756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:43125 (local address 127.24.103.66:58567)
0619 15:04:09.904340 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:09.904360 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:09.904556 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:09.905074 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:09.905078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:09.905103 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:04:09.905362 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.905369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.906903 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.906908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.909976 (+ 3068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.910004 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.911794 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.911803 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.912065 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.912137 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 15:04:09.912247 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 15:04:09.912457 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":921,"threads_started":1}
W20260619 15:04:09.913396 3328 tablet.cc:2369] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:04:09.914431 3014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13475478b6744ece8982259ecf658097" candidate_uuid: "51a7f19cdf1b428a95f6cea123f0a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598d804d838548a3b82835d5e7554fc9" is_pre_election: true
I20260619 15:04:09.915079 3014 raft_consensus.cc:2454] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a7f19cdf1b428a95f6cea123f0a265 in term 0.
I20260619 15:04:09.916044 3082 leader_election.cc:302] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a7f19cdf1b428a95f6cea123f0a265, 598d804d838548a3b82835d5e7554fc9; no voters:
I20260619 15:04:09.916651 3352 raft_consensus.cc:2781] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:04:09.916821 3352 raft_consensus.cc:482] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:04:09.916939 3352 raft_consensus.cc:3037] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:04:09.919844 3352 raft_consensus.cc:504] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598d804d838548a3b82835d5e7554fc9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43125 } } peers { permanent_uuid: "0325e7350d6d4ff39d1d286db2d02264" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43989 } } peers { permanent_uuid: "51a7f19cdf1b428a95f6cea123f0a265" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35371 } }
I20260619 15:04:09.921072 3354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.903795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:43989 (local address 127.24.103.66:55317)
0619 15:04:09.913238 (+ 9443us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:09.913258 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:09.913444 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:09.913971 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:09.913975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:09.913998 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:04:09.914147 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.914159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.916057 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.916065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.918837 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:09.918857 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.920450 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.920455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:09.920602 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.920627 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 15:04:09.920699 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 15:04:09.920796 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9346,"thread_start_us":15546,"threads_started":1}
I20260619 15:04:09.921021 3014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13475478b6744ece8982259ecf658097" candidate_uuid: "51a7f19cdf1b428a95f6cea123f0a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598d804d838548a3b82835d5e7554fc9"
I20260619 15:04:09.921154 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.907210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:55317 (local address 127.24.103.67:43989)
0619 15:04:09.907402 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:09.907409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:09.913335 (+ 5926us) server_negotiation.cc:407] Connection header received
0619 15:04:09.913628 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:09.913633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:09.913716 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:09.913782 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:04:09.914322 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.915833 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.919028 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.920268 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.920439 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.920511 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:09.920753 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 15:04:09.920894 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 15:04:09.921228 3014 raft_consensus.cc:3037] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:04:09.922156 3282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13475478b6744ece8982259ecf658097" candidate_uuid: "51a7f19cdf1b428a95f6cea123f0a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0325e7350d6d4ff39d1d286db2d02264" is_pre_election: true
I20260619 15:04:09.922669 3282 raft_consensus.cc:2454] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a7f19cdf1b428a95f6cea123f0a265 in term 0.
I20260619 15:04:09.923167 3352 leader_election.cc:288] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [CANDIDATE]: Term 1 election: Requested vote from peers 598d804d838548a3b82835d5e7554fc9 (127.24.103.65:43125), 0325e7350d6d4ff39d1d286db2d02264 (127.24.103.67:43989)
I20260619 15:04:09.923913 3282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13475478b6744ece8982259ecf658097" candidate_uuid: "51a7f19cdf1b428a95f6cea123f0a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0325e7350d6d4ff39d1d286db2d02264"
I20260619 15:04:09.924142 3014 raft_consensus.cc:2454] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a7f19cdf1b428a95f6cea123f0a265 in term 1.
I20260619 15:04:09.924206 3282 raft_consensus.cc:3037] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:04:09.924592 3082 leader_election.cc:302] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a7f19cdf1b428a95f6cea123f0a265, 598d804d838548a3b82835d5e7554fc9; no voters:
I20260619 15:04:09.924894 3352 raft_consensus.cc:2781] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:04:09.925454 3352 raft_consensus.cc:686] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 1 LEADER]: Becoming Leader. State: Replica: 51a7f19cdf1b428a95f6cea123f0a265, State: Running, Role: LEADER
I20260619 15:04:09.925967 3352 consensus_queue.cc:237] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598d804d838548a3b82835d5e7554fc9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43125 } } peers { permanent_uuid: "0325e7350d6d4ff39d1d286db2d02264" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43989 } } peers { permanent_uuid: "51a7f19cdf1b428a95f6cea123f0a265" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35371 } }
I20260619 15:04:09.926925 3282 raft_consensus.cc:2454] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a7f19cdf1b428a95f6cea123f0a265 in term 1.
I20260619 15:04:09.931398 2869 catalog_manager.cc:5495] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51a7f19cdf1b428a95f6cea123f0a265 (127.24.103.66). New cstate: current_term: 1 leader_uuid: "51a7f19cdf1b428a95f6cea123f0a265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598d804d838548a3b82835d5e7554fc9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0325e7350d6d4ff39d1d286db2d02264" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51a7f19cdf1b428a95f6cea123f0a265" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35371 } health_report { overall_health: HEALTHY } } }
I20260619 15:04:09.956154 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:09.950485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.24.103.66:35371)
0619 15:04:09.950706 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:09.950712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:09.950806 (+ 94us) server_negotiation.cc:407] Connection header received
0619 15:04:09.951129 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:09.951138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:09.951258 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:09.951429 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:04:09.951883 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.952577 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.953894 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:09.954344 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:09.954457 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:09.954961 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:04:09.955532 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:04:09.955645 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:09.955761 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 15:04:09.955825 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 15:04:09.964424 3282 raft_consensus.cc:1261] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264 [term 1 FOLLOWER]: Refusing update from remote peer 51a7f19cdf1b428a95f6cea123f0a265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:04:09.964424 3014 raft_consensus.cc:1261] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9 [term 1 FOLLOWER]: Refusing update from remote peer 51a7f19cdf1b428a95f6cea123f0a265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:04:09.965425 3352 consensus_queue.cc:1035] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598d804d838548a3b82835d5e7554fc9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:04:09.965719 3356 consensus_queue.cc:1035] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0325e7350d6d4ff39d1d286db2d02264" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:04:09.994174 2870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:04:09.994459 2870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:04:09.998075 2870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2f0783ff738477ab244313b1b4bdd7c: Sending DeleteTablet for 3 replicas of tablet 13475478b6744ece8982259ecf658097
I20260619 15:04:09.999338 2994 tablet_service.cc:1514] Processing DeleteTablet for tablet 13475478b6744ece8982259ecf658097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:04:09 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:44716
I20260619 15:04:09.999343 3262 tablet_service.cc:1514] Processing DeleteTablet for tablet 13475478b6744ece8982259ecf658097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:04:09 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:42304
I20260619 15:04:10.000917 3128 tablet_service.cc:1514] Processing DeleteTablet for tablet 13475478b6744ece8982259ecf658097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:04:09 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:50538
I20260619 15:04:10.005076 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 2931
I20260619 15:04:10.006454 3371 tablet_replica.cc:330] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264: stopping tablet replica
I20260619 15:04:10.006454 3372 tablet_replica.cc:330] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265: stopping tablet replica
I20260619 15:04:10.006942 3372 raft_consensus.cc:2229] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:04:10.007615 3372 raft_consensus.cc:2258] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:04:10.011602 3372 ts_tablet_manager.cc:1905] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:04:10.014988 3371 raft_consensus.cc:2229] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:04:10.015594 3371 raft_consensus.cc:2258] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:04:10.016455 3370 tablet_replica.cc:330] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9: stopping tablet replica
I20260619 15:04:10.016882 3370 raft_consensus.cc:2229] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:04:10.017369 3370 raft_consensus.cc:2258] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:04:10.017725 3371 ts_tablet_manager.cc:1905] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:04:10.018550 3372 ts_tablet_manager.cc:1918] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:04:10.018733 3372 log.cc:1196] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/13475478b6744ece8982259ecf658097
I20260619 15:04:10.022429 3371 ts_tablet_manager.cc:1918] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:04:10.022612 3371 log.cc:1196] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/13475478b6744ece8982259ecf658097
I20260619 15:04:10.023422 3371 ts_tablet_manager.cc:1939] T 13475478b6744ece8982259ecf658097 P 0325e7350d6d4ff39d1d286db2d02264: Deleting consensus metadata
I20260619 15:04:10.023591 3372 ts_tablet_manager.cc:1939] T 13475478b6744ece8982259ecf658097 P 51a7f19cdf1b428a95f6cea123f0a265: Deleting consensus metadata
I20260619 15:04:10.024519 3370 ts_tablet_manager.cc:1905] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:04:10.025349 2854 catalog_manager.cc:4841] TS 0325e7350d6d4ff39d1d286db2d02264 (127.24.103.67:43989): tablet 13475478b6744ece8982259ecf658097 (table test-table [id=700b9d413157495c8edfb3fed18d77d2]) successfully deleted
I20260619 15:04:10.025410 2857 catalog_manager.cc:4841] TS 51a7f19cdf1b428a95f6cea123f0a265 (127.24.103.66:35371): tablet 13475478b6744ece8982259ecf658097 (table test-table [id=700b9d413157495c8edfb3fed18d77d2]) successfully deleted
I20260619 15:04:10.031636 3370 ts_tablet_manager.cc:1918] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:04:10.031808 3370 log.cc:1196] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/13475478b6744ece8982259ecf658097
I20260619 15:04:10.032342 3370 ts_tablet_manager.cc:1939] T 13475478b6744ece8982259ecf658097 P 598d804d838548a3b82835d5e7554fc9: Deleting consensus metadata
Jun 19 15:04:10 dist-test-slave-zpbw krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:04:10.034219 2855 catalog_manager.cc:4841] TS 598d804d838548a3b82835d5e7554fc9 (127.24.103.65:43125): tablet 13475478b6744ece8982259ecf658097 (table test-table [id=700b9d413157495c8edfb3fed18d77d2]) successfully deleted
I20260619 15:04:10.040161 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:10.006861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.24.103.65:43125)
0619 15:04:10.007108 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:10.007117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:10.022929 (+ 15812us) server_negotiation.cc:407] Connection header received
0619 15:04:10.025794 (+ 2865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:10.025798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:10.025881 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:10.026010 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:10.027306 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:10.028054 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:10.029446 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:10.029900 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:10.030071 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:10.034166 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:10.034177 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:10.034190 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:10.034228 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:10.037110 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:10.038041 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:10.038054 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:10.038058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:10.038200 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:10.038607 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:10.038610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:10.038611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:10.039074 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:10.039328 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:10.039679 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 15:04:10.039844 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 15:04:10.040879 3054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:10.155185 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 2931
I20260619 15:04:10.173396 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 3065
Jun 19 15:04:10 dist-test-slave-zpbw krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:04:10.189142 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:10.174192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.24.103.66:35371)
0619 15:04:10.174365 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:10.174371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:10.174507 (+ 136us) server_negotiation.cc:407] Connection header received
0619 15:04:10.174718 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:10.174721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:10.174859 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:10.174977 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:10.176180 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:10.176958 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:10.178175 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:10.178589 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:10.178740 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:10.182771 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:10.182832 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:10.182855 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:10.182887 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:10.186180 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:10.186890 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:10.186901 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:10.186909 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:10.187059 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:10.187453 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:10.187457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:10.187462 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:10.187839 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:10.188138 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:10.188426 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 15:04:10.188669 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 15:04:10.189836 3188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:10.301837 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 3065
I20260619 15:04:10.320787 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 3199
Jun 19 15:04:10 dist-test-slave-zpbw krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:04:10.335889 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:10.321547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.24.103.67:43989)
0619 15:04:10.321719 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:10.321724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:10.321741 (+ 17us) server_negotiation.cc:407] Connection header received
0619 15:04:10.321878 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:10.321881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:10.321939 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:10.322036 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:10.323334 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:10.324059 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:10.325349 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:10.325723 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:10.325813 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:10.330317 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:10.330332 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:10.330347 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:10.330387 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:10.333037 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:10.333732 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:10.333739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:10.333741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:10.333874 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:10.334242 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:10.334248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:10.334252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:10.334606 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:10.334932 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:10.335333 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 15:04:10.335540 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:04:10.336541 3322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:10.424470 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 3199
I20260619 15:04:10.440435 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 2838
I20260619 15:04:10.441638 2900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:10.551728 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 2838
2026-06-19T15:04:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4365 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:04:10 dist-test-slave-zpbw krb5kdc[3380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:04:10 dist-test-slave-zpbw krb5kdc[3380](info): set up 2 sockets
Jun 19 15:04:10 dist-test-slave-zpbw krb5kdc[3380](info): commencing operation
krb5kdc: starting...
W20260619 15:04:11.707881 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:04:11 dist-test-slave-zpbw krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:04:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/rangeradmin_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/rangeradmin_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/rangerlookup_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/rangerlookup_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
I20260619 15:04:12.040015 24989 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskn730WE/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/postgres -l logfile start
2026-06-19 15:04:14.413 UTC [3410] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 15:04:14.414 UTC [3410] LOG: listening on IPv4 address "127.24.103.84", port 48099
2026-06-19 15:04:14.417 UTC [3410] LOG: listening on Unix socket "/tmp/.s.PGSQL.48099"
2026-06-19 15:04:14.440 UTC [3413] LOG: database system was shut down at 2026-06-19 15:04:13 UTC
2026-06-19 15:04:14.444 UTC [3410] LOG: database system is ready to accept connections
I20260619 15:04:15.427443 24989 mini_postgres.cc:96] Postgres bound to 48099
2026-06-19 15:04:15.444 UTC [3421] FATAL: database "slave" does not exist
127.24.103.84:48099 - accepting connections
I20260619 15:04:15.445681 24989 mini_ranger.cc:161] Starting Ranger...
I20260619 15:04:15.472405 24989 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 15:04:15.944478 24989 mini_ranger.cc:87] Created ranger Postgres database
I20260619 15:04:15.944604 24989 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-admin
2026-06-19 15:04:16,256 [I] DB FLAVOR :POSTGRES
2026-06-19 15:04:16,257 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 15:04:16,257 [I] Checking connection..
2026-06-19 15:04:16,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 15:04:16,520 [I] Checking connection passed.
2026-06-19 15:04:16,520 [I] --------- Verifying version history table ---------
2026-06-19 15:04:16,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 15:04:16,771 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 15:04:16,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 15:04:17,030 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 15:04:17,031 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 15:04:17,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 15:04:17.283 UTC [3485] WARNING: there is no transaction in progress
2026-06-19 15:04:17,289 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 15:04:17,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 15:04:17,525 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 15:04:17,525 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 15:04:17,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 15:04:17,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 15:04:18,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zpbw.c.gcp-upstream.internal','N') ;"
2026-06-19 15:04:18,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 15:04:18,537 [I] Table x_portal_user does not exist in database ranger
2026-06-19 15:04:18,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 15:04:18,793 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 15:04:18,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 15:04:19,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 15:04:19,312 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 15:04:19,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 15:04:20.278 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.295 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.311 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.327 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.342 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.359 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.412 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.418 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.432 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.442 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.452 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.466 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.475 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.482 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.488 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:20.495 UTC [3593] WARNING: there is no transaction in progress
2026-06-19 15:04:21,065 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 15:04:21,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 15:04:21,292 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 15:04:21,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 15:04:21,534 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 15:04:21,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 15:04:21,801 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 15:04:21,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 15:04:22,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 15:04:22,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zpbw.c.gcp-upstream.internal';"
2026-06-19 15:04:22,547 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 15:04:22,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 15:04:22,796 [I] DB_PATCHES have already been applied
I20260619 15:04:22.802287 24989 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-admin:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskn730WE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 15:04:22.802384 24989 mini_ranger.cc:193] Using host: 127.24.103.84
I20260619 15:04:22.817425 24989 mini_ranger.cc:227] Ranger admin URL: http://127.24.103.84:35245
Jun 19, 2026 3:04:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 3:04:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 3:04:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 3:04:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 3:04:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 3:04:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 3:04:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 15:04:23.981120 24989 mini_ranger.cc:160] Time spent starting Ranger: real 8.535s user 0.000s sys 0.041s
Jun 19, 2026 3:04:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 3:04:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 3:04:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 3:04:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 3:04:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 3:04:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 15:04:45.112495 24989 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/rangerkms_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/rangerkms_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.103.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.103.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 15:04:45 dist-test-slave-zpbw krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881485, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 15:04:45.280121 24989 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 15:04:45.301688 24989 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 15:04:46.109577 24989 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 15:04:46.109768 24989 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-kms
I20260619 15:04:46.109901 24989 mini_ranger_kms.cc:227] Using postgres at 127.24.103.84:48099
2026-06-19 15:04:46,563 [I] DB FLAVOR :POSTGRES
2026-06-19 15:04:46,564 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 15:04:46,564 [I] Checking connection
2026-06-19 15:04:46,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 15:04:46,802 [I] connection success
2026-06-19 15:04:46,802 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 15:04:46,802 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 15:04:46,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 15:04:47,043 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 15:04:47,044 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 15:04:47,044 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 15:04:47,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:48099/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 15:04:47,310 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 15:04:47.766347 24989 mini_ranger_kms.cc:310] Created kms service
I20260619 15:04:47.992425 24989 mini_ranger_kms.cc:326] Created kudu user
I20260619 15:04:48.039541 24989 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 15:04:48 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881485, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:04:48.450878 24989 mini_ranger_kms.cc:384] Added ranger policy
I20260619 15:04:48.451202 24989 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-kms:/tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 15:04:48.451263 24989 mini_ranger_kms.cc:240] Using host: 127.24.103.84
I20260619 15:04:48.468775 24989 mini_ranger_kms.cc:276] Ranger KMS PID: 3773
I20260619 15:04:48.468997 24989 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.103.84:40171
Jun 19, 2026 3:04:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 3:04:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 3:04:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 3:04:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 3:04:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 3:04:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 15:04:50.887144 24989 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.607s user 0.106s sys 0.367s
I20260619 15:04:50.887830 24989 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 15:04:50.888088 24989 mini_ranger_kms.cc:401] 127.24.103.84:40171/kms/v1/keys
Jun 19 15:04:51 dist-test-slave-zpbw krb5kdc[3380](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781881491, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.103.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 15:04:51 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881491, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.103.84@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
Jun 19 15:04:51 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881491, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.103.84@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
Jun 19 15:04:54 dist-test-slave-zpbw krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:04:54.311287 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44537
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:44537
--ranger_config_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:40171/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:04:54.663940 3824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:54.664261 3824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:54.664429 3824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:54.671718 3824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:04:54.671840 3824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:54.671953 3824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:54.672025 3824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:04:54.672068 3824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:04:54.682379 3824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.103.84:40171/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:44537
--ranger_config_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:54.683831 3824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:54.685247 3824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:54.697558 3829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:54.697683 3830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:54.697716 3832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:54.699323 3824 server_base.cc:1034] running on GCE node
I20260619 15:04:54.700158 3824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:54.701683 3824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:54.702889 3824 hybrid_clock.cc:648] HybridClock initialized: now 1781881494702840 us; error 63 us; skew 500 ppm
I20260619 15:04:54.703254 3824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:04:54 dist-test-slave-zpbw krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881494, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:04:54.706512 3824 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:04:54.708069 3824 webserver.cc:460] Webserver started at http://127.24.103.126:34005/ using document root <none> and password file <none>
I20260619 15:04:54.708771 3824 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:54.708912 3824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:54.709220 3824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 15:04:54 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881494, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:04:54.773101 3824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "1f8ff335397d40429d0d8408a4573d6c"
format_stamp: "Formatted at 2026-06-19 15:04:54 on dist-test-slave-zpbw"
server_key: "2dc9c7016aad1521ffdf8e0a6b5e6530"
server_key_iv: "73e7a19f8b4ab95c79da031ab0bc09c5"
server_key_version: "kuduclusterkey@0"
I20260619 15:04:54.774916 3824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "1f8ff335397d40429d0d8408a4573d6c"
format_stamp: "Formatted at 2026-06-19 15:04:54 on dist-test-slave-zpbw"
server_key: "2dc9c7016aad1521ffdf8e0a6b5e6530"
server_key_iv: "73e7a19f8b4ab95c79da031ab0bc09c5"
server_key_version: "kuduclusterkey@0"
I20260619 15:04:54.782307 3824 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260619 15:04:54.802256 3839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:54.804263 3824 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 15:04:54.804504 3824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "1f8ff335397d40429d0d8408a4573d6c"
format_stamp: "Formatted at 2026-06-19 15:04:54 on dist-test-slave-zpbw"
server_key: "2dc9c7016aad1521ffdf8e0a6b5e6530"
server_key_iv: "73e7a19f8b4ab95c79da031ab0bc09c5"
server_key_version: "kuduclusterkey@0"
I20260619 15:04:54.804777 3824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:54.851234 3824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:54.857048 3824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:54.857379 3824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:54.878655 3824 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:44537
I20260619 15:04:54.878650 3890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:44537 every 8 connection(s)
I20260619 15:04:54.880477 3824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:04:54.885425 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 3824
I20260619 15:04:54.885643 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:04:54.886451 24989 external_mini_cluster.cc:1428] Setting key 07e3ed2b40873f0bd5f5a42041744f1a
I20260619 15:04:54.887089 3891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:04:54.904922 3891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c: Bootstrap starting.
I20260619 15:04:54.909251 3891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c: Neither blocks nor log segments found. Creating new log.
I20260619 15:04:54.910542 3891 log.cc:824] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c: Log is configured to *not* fsync() on all Append() calls
I20260619 15:04:54.913538 3891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c: No bootstrap required, opened a new log
Jun 19 15:04:54 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:04:54.924365 3891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8ff335397d40429d0d8408a4573d6c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44537 } }
I20260619 15:04:54.924954 3891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:04:54.925060 3891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8ff335397d40429d0d8408a4573d6c, State: Initialized, Role: FOLLOWER
I20260619 15:04:54.925762 3891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8ff335397d40429d0d8408a4573d6c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44537 } }
I20260619 15:04:54.926061 3891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:04:54.926262 3891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:04:54.926470 3891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:04:54.929795 3891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8ff335397d40429d0d8408a4573d6c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44537 } }
I20260619 15:04:54.930419 3891 leader_election.cc:302] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f8ff335397d40429d0d8408a4573d6c; no voters:
I20260619 15:04:54.930716 3894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:54.890624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.24.103.126:44537)
0619 15:04:54.903390 (+ 12766us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:54.903419 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:54.903482 (+ 63us) server_negotiation.cc:407] Connection header received
0619 15:04:54.905997 (+ 2515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:54.906030 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:54.906609 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:54.907120 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:54.909234 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:54.911376 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:54.913415 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:54.914464 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:54.914651 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:54.919865 (+ 5214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:54.919877 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:54.919903 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:54.919952 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:54.924521 (+ 4569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:54.925519 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:54.925529 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:54.925539 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:54.925754 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:54.926286 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:54.926291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:54.926294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:54.927122 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:54.927438 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:54.928794 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0619 15:04:54.929226 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12492,"thread_start_us":12203,"threads_started":1}
I20260619 15:04:54.931236 3891 leader_election.cc:288] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:04:54.931434 3896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:04:54.932863 3896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [term 1 LEADER]: Becoming Leader. State: Replica: 1f8ff335397d40429d0d8408a4573d6c, State: Running, Role: LEADER
I20260619 15:04:54.933568 3891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:04:54.933493 3896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8ff335397d40429d0d8408a4573d6c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44537 } }
I20260619 15:04:54.938686 3898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f8ff335397d40429d0d8408a4573d6c. Latest consensus state: current_term: 1 leader_uuid: "1f8ff335397d40429d0d8408a4573d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8ff335397d40429d0d8408a4573d6c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44537 } } }
I20260619 15:04:54.939101 3898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [sys.catalog]: This master's current role is: LEADER
I20260619 15:04:54.938977 3897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f8ff335397d40429d0d8408a4573d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8ff335397d40429d0d8408a4573d6c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44537 } } }
I20260619 15:04:54.939195 3897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c [sys.catalog]: This master's current role is: LEADER
I20260619 15:04:54.941648 3906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:04:54.947321 3906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:04:54.947962 3891 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 15:04:54.964874 3906 catalog_manager.cc:1347] Generated new cluster ID: f360d636dfdc4f60a14388fce360f7bd
I20260619 15:04:54.965054 3906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:04:54.989315 3906 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:04:54.991518 3906 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:04:55.006613 3906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c: Generated new TSK 0
I20260619 15:04:55.008632 3906 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:04:55.806330 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:40171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44537
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 15:04:56 dist-test-slave-zpbw krb5kdc[3380](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781881496, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 15:04:56.274129 3936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:56.274474 3936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:56.274641 3936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:56.282640 3936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:56.282861 3936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:56.283072 3936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:04:56.296026 3936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.103.84:40171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:56.297626 3936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:56.299532 3936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:56.325291 3946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:56.325991 3947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:56.333444 3936 server_base.cc:1034] running on GCE node
W20260619 15:04:56.333865 3949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:56.334693 3936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:56.335987 3936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:56.337296 3936 hybrid_clock.cc:648] HybridClock initialized: now 1781881496337218 us; error 73 us; skew 500 ppm
I20260619 15:04:56.337832 3936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:04:56 dist-test-slave-zpbw krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881496, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:04:56.342119 3936 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:04:56.343958 3936 webserver.cc:460] Webserver started at http://127.24.103.65:45283/ using document root <none> and password file <none>
I20260619 15:04:56.344732 3936 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:56.344885 3936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:56.345206 3936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 15:04:56 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881496, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:04:56.364902 3936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "bfbc366cdb934ccda6bd64d44bd7fa80"
format_stamp: "Formatted at 2026-06-19 15:04:56 on dist-test-slave-zpbw"
server_key: "3ab6c126c53e5b2e5c6ba174a0bbc55a"
server_key_iv: "18c90dc98a56e4c4d2c46a6faa43b200"
server_key_version: "kuduclusterkey@0"
I20260619 15:04:56.366233 3936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "bfbc366cdb934ccda6bd64d44bd7fa80"
format_stamp: "Formatted at 2026-06-19 15:04:56 on dist-test-slave-zpbw"
server_key: "3ab6c126c53e5b2e5c6ba174a0bbc55a"
server_key_iv: "18c90dc98a56e4c4d2c46a6faa43b200"
server_key_version: "kuduclusterkey@0"
I20260619 15:04:56.388022 3936 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.011s sys 0.006s
I20260619 15:04:56.418107 3956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:56.420176 3936 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.005s
I20260619 15:04:56.420468 3936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "bfbc366cdb934ccda6bd64d44bd7fa80"
format_stamp: "Formatted at 2026-06-19 15:04:56 on dist-test-slave-zpbw"
server_key: "3ab6c126c53e5b2e5c6ba174a0bbc55a"
server_key_iv: "18c90dc98a56e4c4d2c46a6faa43b200"
server_key_version: "kuduclusterkey@0"
I20260619 15:04:56.420799 3936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:56.469813 3936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:56.475178 3936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:56.475600 3936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:56.476775 3936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:04:56.484787 3936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:04:56.484930 3936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:56.485081 3936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:04:56.485163 3936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:56.559381 3936 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:37817
I20260619 15:04:56.559394 4068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:37817 every 8 connection(s)
I20260619 15:04:56.560911 3936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:04:56.568051 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 3936
I20260619 15:04:56.568272 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:04:56.569093 24989 external_mini_cluster.cc:1428] Setting key 109ceb0cef14710476418b5e8a91ef70
Jun 19 15:04:56 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881496, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:04:56.605319 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:56.572789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44537 (local address 127.24.103.65:42209)
0619 15:04:56.581305 (+ 8516us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:56.581363 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:56.584297 (+ 2934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:56.586071 (+ 1774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:56.586085 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:56.586591 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:56.587300 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:56.587318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:56.588887 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:56.588892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:56.591622 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:56.591638 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:56.592592 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:56.592602 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:56.592810 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:56.593524 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:04:56.593548 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:56.596100 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:04:56.600173 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:04:56.600183 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:04:56.600202 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:04:56.600831 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:04:56.601518 (+ 687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:04:56.601523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:04:56.601526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:04:56.601823 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:04:56.602527 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:04:56.602536 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:04:56.603029 (+ 493us) client_negotiation.cc:769] Sending connection context
0619 15:04:56.603347 (+ 318us) client_negotiation.cc:240] Negotiation successful
0619 15:04:56.603923 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8161,"thread_start_us":7899,"threads_started":1}
I20260619 15:04:56.605870 4072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:56.573437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:42209 (local address 127.24.103.126:44537)
0619 15:04:56.585177 (+ 11740us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:56.585186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:56.585212 (+ 26us) server_negotiation.cc:407] Connection header received
0619 15:04:56.585286 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:56.585293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:56.585412 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:56.585541 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:56.587487 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:56.588725 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:56.591813 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:56.592224 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:56.592372 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:56.596397 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:56.596404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:56.596409 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:56.596451 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:56.599983 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:56.600983 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:56.600988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:56.600993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:56.601381 (+ 388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:56.601959 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:56.601965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:56.601970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:56.602311 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:56.602446 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:56.605251 (+ 2805us) server_negotiation.cc:299] Negotiation successful
0619 15:04:56.605501 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11570,"thread_start_us":12570,"threads_started":1}
I20260619 15:04:56.608042 4069 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44537
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260619 15:04:56.608439 4069 heartbeater.cc:464] Registering TS with master...
I20260619 15:04:56.609383 4069 heartbeater.cc:511] Master 127.24.103.126:44537 requested a full tablet report, sending...
I20260619 15:04:56.612074 3856 ts_manager.cc:194] Registered new tserver with Master: bfbc366cdb934ccda6bd64d44bd7fa80 (127.24.103.65:37817)
I20260619 15:04:56.614430 3856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:42209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Jun 19 15:04:56 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881496, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:04:56.657356 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:40171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44537
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 15:04:56 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881496, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
W20260619 15:04:57.154990 4075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:57.155480 4075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:57.155651 4075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:57.167615 4075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:57.167866 4075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:57.168112 4075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:04:57.183485 4075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.103.84:40171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:57.185235 4075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:57.187004 4075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:57.221367 4083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:57.222153 4085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:57.222627 4082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:57.232497 4075 server_base.cc:1034] running on GCE node
I20260619 15:04:57.233608 4075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:57.235661 4075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:57.239039 4075 hybrid_clock.cc:648] HybridClock initialized: now 1781881497238921 us; error 108 us; skew 500 ppm
I20260619 15:04:57.239450 4075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:04:57 dist-test-slave-zpbw krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881497, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:04:57.244282 4075 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:04:57.259078 4075 webserver.cc:460] Webserver started at http://127.24.103.66:33973/ using document root <none> and password file <none>
I20260619 15:04:57.260196 4075 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:57.260399 4075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:57.260816 4075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 15:04:57 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881497, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:04:57.285351 4075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5"
format_stamp: "Formatted at 2026-06-19 15:04:57 on dist-test-slave-zpbw"
server_key: "8376176dc2d503e9be777358fa4f3834"
server_key_iv: "2da8c53975a3237e976604ee95152962"
server_key_version: "kuduclusterkey@0"
I20260619 15:04:57.286438 4075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5"
format_stamp: "Formatted at 2026-06-19 15:04:57 on dist-test-slave-zpbw"
server_key: "8376176dc2d503e9be777358fa4f3834"
server_key_iv: "2da8c53975a3237e976604ee95152962"
server_key_version: "kuduclusterkey@0"
I20260619 15:04:57.293716 4075 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.001s
I20260619 15:04:57.323190 4095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:57.324966 4075 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.006s
I20260619 15:04:57.325212 4075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5"
format_stamp: "Formatted at 2026-06-19 15:04:57 on dist-test-slave-zpbw"
server_key: "8376176dc2d503e9be777358fa4f3834"
server_key_iv: "2da8c53975a3237e976604ee95152962"
server_key_version: "kuduclusterkey@0"
I20260619 15:04:57.325460 4075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:57.341363 4075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:57.345675 4075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:57.346000 4075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:57.347110 4075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:04:57.348551 4075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:04:57.348652 4075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:57.348793 4075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:04:57.348865 4075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:57.392580 4075 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:46865
I20260619 15:04:57.392558 4211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:46865 every 8 connection(s)
I20260619 15:04:57.394361 4075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:04:57.404166 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 4075
I20260619 15:04:57.404357 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:04:57.405022 24989 external_mini_cluster.cc:1428] Setting key a95c3d47e8ff29c3945d5972d065121e
Jun 19 15:04:57 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881497, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:04:57.417186 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:57.397283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:50165 (local address 127.24.103.126:44537)
0619 15:04:57.397808 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:57.397815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:57.398854 (+ 1039us) server_negotiation.cc:407] Connection header received
0619 15:04:57.401551 (+ 2697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:57.401557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:57.401655 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:57.401783 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:57.403388 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:57.404531 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:57.406684 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:57.407186 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:57.407330 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:57.410736 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:57.410741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:57.410744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:57.410833 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:57.413682 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:57.414444 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:57.414450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:57.414456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:57.414577 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:57.415123 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:57.415144 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:57.415148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:57.415441 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:57.415600 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:57.416584 (+ 984us) server_negotiation.cc:299] Negotiation successful
0619 15:04:57.416808 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":250,"threads_started":1}
I20260619 15:04:57.418195 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:57.397859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44537 (local address 127.24.103.66:50165)
0619 15:04:57.398670 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:57.398722 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:57.401241 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:57.401993 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:57.402002 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:57.402485 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:57.403192 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:57.403210 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:57.404715 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:57.404721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:57.406519 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:57.406532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:57.407419 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:57.407427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:57.407667 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:57.408395 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:04:57.408418 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:57.410497 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:04:57.413841 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:04:57.413851 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:04:57.413870 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:04:57.414306 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:04:57.414700 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:04:57.414704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:04:57.414708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:04:57.415007 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:04:57.415572 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:04:57.415580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:04:57.416078 (+ 498us) client_negotiation.cc:769] Sending connection context
0619 15:04:57.416414 (+ 336us) client_negotiation.cc:240] Negotiation successful
0619 15:04:57.417034 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":333,"threads_started":1}
I20260619 15:04:57.419713 4212 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44537
I20260619 15:04:57.420050 4212 heartbeater.cc:464] Registering TS with master...
I20260619 15:04:57.420800 4212 heartbeater.cc:511] Master 127.24.103.126:44537 requested a full tablet report, sending...
I20260619 15:04:57.422365 3856 ts_manager.cc:194] Registered new tserver with Master: 1e4cfed7941047e1a811ef3b2f8d5ce5 (127.24.103.66:46865)
I20260619 15:04:57.423275 3856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:50165
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:04:57.465950 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:40171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44537
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:04:57.618690 4069 heartbeater.cc:503] Master 127.24.103.126:44537 was elected leader, sending a full tablet report...
W20260619 15:04:57.819482 4218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:04:57.819834 4218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:04:57.820008 4218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:04:57.828343 4218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:04:57.828528 4218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:04:57.828717 4218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:04:57.839890 4218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.103.84:40171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:04:57.841302 4218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:04:57.842700 4218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:04:57.857559 4223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:57.857596 4224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:04:57.859151 4226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:04:57.859591 4218 server_base.cc:1034] running on GCE node
I20260619 15:04:57.860468 4218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:04:57.861743 4218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:04:57.863065 4218 hybrid_clock.cc:648] HybridClock initialized: now 1781881497863013 us; error 43 us; skew 500 ppm
I20260619 15:04:57.863402 4218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:04:57 dist-test-slave-zpbw krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881497, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:04:57.866561 4218 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:04:57.868162 4218 webserver.cc:460] Webserver started at http://127.24.103.67:37539/ using document root <none> and password file <none>
I20260619 15:04:57.868872 4218 fs_manager.cc:359] Metadata directory not provided
I20260619 15:04:57.869002 4218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:04:57.869305 4218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 15:04:57 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881497, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:04:57.885829 4218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "52259d4dca0047f29eed7fcf9cb958fa"
format_stamp: "Formatted at 2026-06-19 15:04:57 on dist-test-slave-zpbw"
server_key: "b1a33bb1f1630bf09177d02ae038a7f7"
server_key_iv: "eab0dc23afb1ce44e78878b5c5c36a75"
server_key_version: "kuduclusterkey@0"
I20260619 15:04:57.886689 4218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "52259d4dca0047f29eed7fcf9cb958fa"
format_stamp: "Formatted at 2026-06-19 15:04:57 on dist-test-slave-zpbw"
server_key: "b1a33bb1f1630bf09177d02ae038a7f7"
server_key_iv: "eab0dc23afb1ce44e78878b5c5c36a75"
server_key_version: "kuduclusterkey@0"
I20260619 15:04:57.892083 4218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 15:04:57.908289 4233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:57.909761 4218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 15:04:57.909971 4218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "52259d4dca0047f29eed7fcf9cb958fa"
format_stamp: "Formatted at 2026-06-19 15:04:57 on dist-test-slave-zpbw"
server_key: "b1a33bb1f1630bf09177d02ae038a7f7"
server_key_iv: "eab0dc23afb1ce44e78878b5c5c36a75"
server_key_version: "kuduclusterkey@0"
I20260619 15:04:57.910202 4218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:04:57.944057 4218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:04:57.947924 4218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:04:57.948254 4218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:04:57.949229 4218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:04:57.950601 4218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:04:57.950672 4218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:57.950842 4218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:04:57.950894 4218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:04:57.986196 4218 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:45181
I20260619 15:04:57.986238 4345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:45181 every 8 connection(s)
I20260619 15:04:57.987474 4218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:04:57.992491 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 4218
I20260619 15:04:57.992687 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:04:57.993170 24989 external_mini_cluster.cc:1428] Setting key 9b89119bdb4921dabb5dfa00ca128ddd
Jun 19 15:04:57 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881497, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:04:58.007032 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:57.990367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:60813 (local address 127.24.103.126:44537)
0619 15:04:57.990822 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:57.990831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:57.991872 (+ 1041us) server_negotiation.cc:407] Connection header received
0619 15:04:57.993791 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:57.993797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:57.993887 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:57.994014 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:57.995492 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:57.996226 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:57.997762 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:57.998183 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:57.998318 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.001531 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:58.001535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:58.001538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:58.001568 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:58.004023 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:58.004737 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:58.004747 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:58.004749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:58.004870 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:58.005285 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:58.005290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:58.005293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:58.005557 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:58.005726 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:58.006376 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 15:04:58.006589 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":213,"threads_started":1}
I20260619 15:04:58.007983 4349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:57.991039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44537 (local address 127.24.103.67:60813)
0619 15:04:57.991705 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:57.991743 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:57.993505 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:57.994246 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:57.994254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:57.994655 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:04:57.995334 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:57.995351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:57.996381 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:57.996386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:57.997617 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:57.997628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:57.998395 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:57.998402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:57.998632 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:57.999322 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:04:57.999344 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:04:58.001306 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:04:58.004171 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:04:58.004180 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:04:58.004197 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:04:58.004611 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:04:58.004978 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:04:58.004985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:04:58.004988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:04:58.005170 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:04:58.005700 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:04:58.005708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:04:58.006021 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 15:04:58.006283 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 15:04:58.006794 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":258,"threads_started":1}
I20260619 15:04:58.009387 4346 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44537
I20260619 15:04:58.009707 4346 heartbeater.cc:464] Registering TS with master...
I20260619 15:04:58.010910 4346 heartbeater.cc:511] Master 127.24.103.126:44537 requested a full tablet report, sending...
I20260619 15:04:58.012293 3856 ts_manager.cc:194] Registered new tserver with Master: 52259d4dca0047f29eed7fcf9cb958fa (127.24.103.67:45181)
I20260619 15:04:58.013080 3856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:60813
I20260619 15:04:58.021035 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:04:58.039834 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.025783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.24.103.126:44537)
0619 15:04:58.025987 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:58.025995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:58.026390 (+ 395us) server_negotiation.cc:407] Connection header received
0619 15:04:58.026600 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:58.026606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:58.026703 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:58.026857 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:58.028477 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.029272 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.030810 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.031264 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.031403 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.033900 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:58.033911 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:58.033915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:58.033947 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:58.036995 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:58.037677 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:58.037684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:58.037688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:58.037806 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:58.038251 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:58.038254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:58.038256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:58.038576 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:58.038862 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:58.039233 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 15:04:58.039495 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 15:04:58.044096 3856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:04:58.051247 3856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:04:58.081835 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.064495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37817 (local address 127.0.0.1:47096)
0619 15:04:58.065800 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:58.065847 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:58.066038 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:58.068837 (+ 2799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:58.068848 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:58.068900 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:04:58.069467 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:58.069482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.073864 (+ 4382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.073873 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:58.077257 (+ 3384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:58.077281 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.080532 (+ 3251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.080539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:58.080746 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.080803 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 15:04:58.080908 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 15:04:58.081511 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":1824,"threads_started":1}
I20260619 15:04:58.081996 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.064619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.24.103.65:37817)
0619 15:04:58.066687 (+ 2068us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:58.066700 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:58.066745 (+ 45us) server_negotiation.cc:407] Connection header received
0619 15:04:58.066901 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:58.066910 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:58.068539 (+ 1629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:58.069821 (+ 1282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:04:58.070092 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.073640 (+ 3548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.078830 (+ 5190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.080293 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.081001 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.081368 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:58.081456 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 15:04:58.081617 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1883,"thread_start_us":1724,"threads_started":1}
I20260619 15:04:58.086581 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.064929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:45181 (local address 127.0.0.1:34760)
0619 15:04:58.066570 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:58.066589 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:58.072921 (+ 6332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:58.078812 (+ 5891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:58.078823 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:58.078856 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:04:58.079748 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:58.079762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.081798 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.081805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:58.084006 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:58.084019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.085814 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.085819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:58.085987 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.086019 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 15:04:58.086103 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 15:04:58.086241 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1515,"thread_start_us":374,"threads_started":1}
I20260619 15:04:58.086704 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.065380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.24.103.67:45181)
0619 15:04:58.078131 (+ 12751us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:58.078140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:58.078176 (+ 36us) server_negotiation.cc:407] Connection header received
0619 15:04:58.078289 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:58.078297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:58.078582 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:58.078827 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:04:58.080109 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.081657 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.084188 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.085579 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.085707 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.085929 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:58.086187 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 15:04:58.086344 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12528,"thread_start_us":13672,"threads_started":1}
I20260619 15:04:58.086521 4004 tablet_service.cc:1467] Processing CreateTablet for tablet 3164dfb6423d4119a506958b5249f840 (DEFAULT_TABLE table=test-table [id=18d93d80421c44adb38562ceeb7825f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:04:58.088672 4004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3164dfb6423d4119a506958b5249f840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:04:58.088724 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.066988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55740 (local address 127.24.103.66:46865)
0619 15:04:58.080303 (+ 13315us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:58.080312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:58.080346 (+ 34us) server_negotiation.cc:407] Connection header received
0619 15:04:58.080426 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:58.080432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:58.080622 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:58.080841 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:04:58.082650 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.084190 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.085720 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.087251 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.087393 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.087558 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:58.088276 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 15:04:58.088439 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13172,"thread_start_us":13082,"threads_started":1}
I20260619 15:04:58.088621 4363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.066722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:46865 (local address 127.0.0.1:55740)
0619 15:04:58.079235 (+ 12513us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:58.079260 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:58.079817 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:58.080808 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:58.080813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:58.080840 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:04:58.082437 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:58.082449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.084322 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.084330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:58.085565 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:58.085575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.087465 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.087471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:58.087629 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.087666 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 15:04:58.088157 (+ 491us) client_negotiation.cc:240] Negotiation successful
0619 15:04:58.088301 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12364,"thread_start_us":13118,"threads_started":1}
I20260619 15:04:58.090907 4281 tablet_service.cc:1467] Processing CreateTablet for tablet 3164dfb6423d4119a506958b5249f840 (DEFAULT_TABLE table=test-table [id=18d93d80421c44adb38562ceeb7825f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:04:58.092481 4281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3164dfb6423d4119a506958b5249f840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:04:58.092715 4147 tablet_service.cc:1467] Processing CreateTablet for tablet 3164dfb6423d4119a506958b5249f840 (DEFAULT_TABLE table=test-table [id=18d93d80421c44adb38562ceeb7825f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:04:58.094038 4147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3164dfb6423d4119a506958b5249f840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:04:58.100666 4365 tablet_bootstrap.cc:492] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80: Bootstrap starting.
I20260619 15:04:58.104096 4365 tablet_bootstrap.cc:654] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80: Neither blocks nor log segments found. Creating new log.
I20260619 15:04:58.105314 4365 log.cc:824] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80: Log is configured to *not* fsync() on all Append() calls
I20260619 15:04:58.108510 4367 tablet_bootstrap.cc:492] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa: Bootstrap starting.
I20260619 15:04:58.109845 4365 tablet_bootstrap.cc:492] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80: No bootstrap required, opened a new log
I20260619 15:04:58.110225 4365 ts_tablet_manager.cc:1397] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 15:04:58.110350 4368 tablet_bootstrap.cc:492] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5: Bootstrap starting.
I20260619 15:04:58.112733 4367 tablet_bootstrap.cc:654] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa: Neither blocks nor log segments found. Creating new log.
I20260619 15:04:58.113517 4368 tablet_bootstrap.cc:654] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5: Neither blocks nor log segments found. Creating new log.
I20260619 15:04:58.114429 4367 log.cc:824] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa: Log is configured to *not* fsync() on all Append() calls
I20260619 15:04:58.114616 4368 log.cc:824] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5: Log is configured to *not* fsync() on all Append() calls
I20260619 15:04:58.117775 4367 tablet_bootstrap.cc:492] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa: No bootstrap required, opened a new log
I20260619 15:04:58.118057 4367 ts_tablet_manager.cc:1397] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260619 15:04:58.118314 4368 tablet_bootstrap.cc:492] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5: No bootstrap required, opened a new log
I20260619 15:04:58.118590 4368 ts_tablet_manager.cc:1397] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 15:04:58.122325 4365 raft_consensus.cc:348] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37817 } } peers { permanent_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46865 } } peers { permanent_uuid: "52259d4dca0047f29eed7fcf9cb958fa" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45181 } }
I20260619 15:04:58.122872 4365 raft_consensus.cc:374] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:04:58.123008 4365 raft_consensus.cc:729] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfbc366cdb934ccda6bd64d44bd7fa80, State: Initialized, Role: FOLLOWER
I20260619 15:04:58.123677 4365 consensus_queue.cc:260] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37817 } } peers { permanent_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46865 } } peers { permanent_uuid: "52259d4dca0047f29eed7fcf9cb958fa" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45181 } }
I20260619 15:04:58.125559 4365 ts_tablet_manager.cc:1428] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260619 15:04:58.127138 4367 raft_consensus.cc:348] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37817 } } peers { permanent_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46865 } } peers { permanent_uuid: "52259d4dca0047f29eed7fcf9cb958fa" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45181 } }
I20260619 15:04:58.127580 4367 raft_consensus.cc:374] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:04:58.127683 4367 raft_consensus.cc:729] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52259d4dca0047f29eed7fcf9cb958fa, State: Initialized, Role: FOLLOWER
I20260619 15:04:58.128330 4367 consensus_queue.cc:260] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37817 } } peers { permanent_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46865 } } peers { permanent_uuid: "52259d4dca0047f29eed7fcf9cb958fa" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45181 } }
I20260619 15:04:58.128764 4368 raft_consensus.cc:348] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37817 } } peers { permanent_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46865 } } peers { permanent_uuid: "52259d4dca0047f29eed7fcf9cb958fa" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45181 } }
I20260619 15:04:58.129220 4368 raft_consensus.cc:374] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:04:58.129635 4346 heartbeater.cc:503] Master 127.24.103.126:44537 was elected leader, sending a full tablet report...
I20260619 15:04:58.130242 4367 ts_tablet_manager.cc:1428] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260619 15:04:58.129379 4368 raft_consensus.cc:729] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4cfed7941047e1a811ef3b2f8d5ce5, State: Initialized, Role: FOLLOWER
I20260619 15:04:58.131301 4368 consensus_queue.cc:260] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37817 } } peers { permanent_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46865 } } peers { permanent_uuid: "52259d4dca0047f29eed7fcf9cb958fa" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45181 } }
I20260619 15:04:58.132863 4212 heartbeater.cc:503] Master 127.24.103.126:44537 was elected leader, sending a full tablet report...
I20260619 15:04:58.133562 4368 ts_tablet_manager.cc:1428] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260619 15:04:58.146530 4213 tablet.cc:2369] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:04:58.239428 4347 tablet.cc:2369] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:04:58.260417 4371 raft_consensus.cc:482] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:04:58.260787 4371 raft_consensus.cc:504] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37817 } } peers { permanent_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46865 } } peers { permanent_uuid: "52259d4dca0047f29eed7fcf9cb958fa" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45181 } }
I20260619 15:04:58.262547 4371 leader_election.cc:288] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e4cfed7941047e1a811ef3b2f8d5ce5 (127.24.103.66:46865), 52259d4dca0047f29eed7fcf9cb958fa (127.24.103.67:45181)
I20260619 15:04:58.270073 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.262759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:46865 (local address 127.24.103.65:40057)
0619 15:04:58.263697 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:58.263717 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:58.263897 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:58.264340 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:58.264343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:58.264368 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:04:58.264782 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:58.264797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.265965 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.265971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:58.267830 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:58.267842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.269195 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.269201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:58.269397 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.269445 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 15:04:58.269545 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 15:04:58.269693 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":340,"threads_started":1}
I20260619 15:04:58.270529 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.263121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:40057 (local address 127.24.103.66:46865)
0619 15:04:58.263336 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:58.263344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:58.263810 (+ 466us) server_negotiation.cc:407] Connection header received
0619 15:04:58.264098 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:58.264104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:58.264187 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:58.264293 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:04:58.264943 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.265840 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.268006 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.269002 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.269113 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.269163 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:58.269601 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 15:04:58.269851 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 15:04:58.270711 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.263157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:42069 (local address 127.24.103.67:45181)
0619 15:04:58.263332 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:58.263336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:58.264467 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 15:04:58.264666 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:58.264674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:58.264756 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:58.264857 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:04:58.265435 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.266278 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.268633 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.269603 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.269732 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.269798 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:58.270333 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 15:04:58.270468 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 15:04:58.271356 4375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.263690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:45181 (local address 127.24.103.65:42069)
0619 15:04:58.264349 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 15:04:58.264367 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:04:58.264527 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:04:58.264898 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:04:58.264903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:04:58.264926 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:04:58.265283 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:58.265295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.266413 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.266417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:58.268460 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:04:58.268474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.269956 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.269961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:04:58.270139 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.270174 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 15:04:58.270891 (+ 717us) client_negotiation.cc:240] Negotiation successful
0619 15:04:58.271020 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":381,"threads_started":1}
I20260619 15:04:58.271674 4167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3164dfb6423d4119a506958b5249f840" candidate_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5" is_pre_election: true
I20260619 15:04:58.272300 4167 raft_consensus.cc:2454] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfbc366cdb934ccda6bd64d44bd7fa80 in term 0.
I20260619 15:04:58.272559 4301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3164dfb6423d4119a506958b5249f840" candidate_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52259d4dca0047f29eed7fcf9cb958fa" is_pre_election: true
I20260619 15:04:58.273015 4301 raft_consensus.cc:2454] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfbc366cdb934ccda6bd64d44bd7fa80 in term 0.
I20260619 15:04:58.273015 3958 leader_election.cc:302] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4cfed7941047e1a811ef3b2f8d5ce5, bfbc366cdb934ccda6bd64d44bd7fa80; no voters:
I20260619 15:04:58.273458 4371 raft_consensus.cc:2781] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:04:58.273618 4371 raft_consensus.cc:482] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:04:58.273754 4371 raft_consensus.cc:3037] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:04:58.276036 4371 raft_consensus.cc:504] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37817 } } peers { permanent_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46865 } } peers { permanent_uuid: "52259d4dca0047f29eed7fcf9cb958fa" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45181 } }
I20260619 15:04:58.276624 4371 leader_election.cc:288] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [CANDIDATE]: Term 1 election: Requested vote from peers 1e4cfed7941047e1a811ef3b2f8d5ce5 (127.24.103.66:46865), 52259d4dca0047f29eed7fcf9cb958fa (127.24.103.67:45181)
I20260619 15:04:58.277130 4167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3164dfb6423d4119a506958b5249f840" candidate_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5"
I20260619 15:04:58.277130 4301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3164dfb6423d4119a506958b5249f840" candidate_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52259d4dca0047f29eed7fcf9cb958fa"
I20260619 15:04:58.277328 4301 raft_consensus.cc:3037] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:04:58.277328 4167 raft_consensus.cc:3037] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:04:58.280494 4167 raft_consensus.cc:2454] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfbc366cdb934ccda6bd64d44bd7fa80 in term 1.
I20260619 15:04:58.280947 4301 raft_consensus.cc:2454] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfbc366cdb934ccda6bd64d44bd7fa80 in term 1.
I20260619 15:04:58.281447 3957 leader_election.cc:302] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52259d4dca0047f29eed7fcf9cb958fa, bfbc366cdb934ccda6bd64d44bd7fa80; no voters:
I20260619 15:04:58.281750 4371 raft_consensus.cc:2781] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:04:58.282186 4371 raft_consensus.cc:686] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 1 LEADER]: Becoming Leader. State: Replica: bfbc366cdb934ccda6bd64d44bd7fa80, State: Running, Role: LEADER
I20260619 15:04:58.282667 4371 consensus_queue.cc:237] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37817 } } peers { permanent_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46865 } } peers { permanent_uuid: "52259d4dca0047f29eed7fcf9cb958fa" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45181 } }
I20260619 15:04:58.287050 3855 catalog_manager.cc:5495] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfbc366cdb934ccda6bd64d44bd7fa80 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbc366cdb934ccda6bd64d44bd7fa80" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52259d4dca0047f29eed7fcf9cb958fa" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45181 } health_report { overall_health: UNKNOWN } } }
I20260619 15:04:58.317795 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.312081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.24.103.65:37817)
0619 15:04:58.312303 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:58.312308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:58.312468 (+ 160us) server_negotiation.cc:407] Connection header received
0619 15:04:58.312709 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:58.312712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:58.312789 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:58.312911 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:04:58.313233 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.314013 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.315655 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.316101 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.316195 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.316706 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:04:58.317249 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:04:58.317358 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:58.317434 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 15:04:58.317502 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 15:04:58.320067 4070 tablet.cc:2369] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:04:58.326880 4301 raft_consensus.cc:1261] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa [term 1 FOLLOWER]: Refusing update from remote peer bfbc366cdb934ccda6bd64d44bd7fa80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:04:58.326903 4167 raft_consensus.cc:1261] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5 [term 1 FOLLOWER]: Refusing update from remote peer bfbc366cdb934ccda6bd64d44bd7fa80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:04:58.327900 4376 consensus_queue.cc:1035] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52259d4dca0047f29eed7fcf9cb958fa" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:04:58.328382 4371 consensus_queue.cc:1035] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4cfed7941047e1a811ef3b2f8d5ce5" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:04:58.340696 4379 mvcc.cc:205] Tried to move back new op lower bound from 7298586617136381952 to 7298586616969416704. Current Snapshot: MvccSnapshot[applied={T|T < 7298586617136381952}]
I20260619 15:04:58.360040 3855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:04:58.360333 3855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:04:58.364332 3855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f8ff335397d40429d0d8408a4573d6c: Sending DeleteTablet for 3 replicas of tablet 3164dfb6423d4119a506958b5249f840
I20260619 15:04:58.365669 4281 tablet_service.cc:1514] Processing DeleteTablet for tablet 3164dfb6423d4119a506958b5249f840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:04:58 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:34760
I20260619 15:04:58.365727 4004 tablet_service.cc:1514] Processing DeleteTablet for tablet 3164dfb6423d4119a506958b5249f840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:04:58 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:47096
I20260619 15:04:58.365780 4147 tablet_service.cc:1514] Processing DeleteTablet for tablet 3164dfb6423d4119a506958b5249f840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:04:58 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:55740
I20260619 15:04:58.369231 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 3936
I20260619 15:04:58.376185 4391 tablet_replica.cc:330] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5: stopping tablet replica
I20260619 15:04:58.377110 4391 raft_consensus.cc:2229] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:04:58.377993 4391 raft_consensus.cc:2258] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:04:58.379117 4390 tablet_replica.cc:330] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80: stopping tablet replica
I20260619 15:04:58.379570 4389 tablet_replica.cc:330] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa: stopping tablet replica
I20260619 15:04:58.379873 4390 raft_consensus.cc:2229] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:04:58.380096 4389 raft_consensus.cc:2229] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:04:58.380585 4390 raft_consensus.cc:2258] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:04:58.380724 4389 raft_consensus.cc:2258] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:04:58.383201 4389 ts_tablet_manager.cc:1905] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:04:58.383219 4390 ts_tablet_manager.cc:1905] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:04:58.383309 4391 ts_tablet_manager.cc:1905] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:04:58.390304 4389 ts_tablet_manager.cc:1918] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:04:58.390488 4389 log.cc:1196] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/3164dfb6423d4119a506958b5249f840
I20260619 15:04:58.391289 4390 ts_tablet_manager.cc:1918] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:04:58.391541 4389 ts_tablet_manager.cc:1939] T 3164dfb6423d4119a506958b5249f840 P 52259d4dca0047f29eed7fcf9cb958fa: Deleting consensus metadata
I20260619 15:04:58.391677 4390 log.cc:1196] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/3164dfb6423d4119a506958b5249f840
I20260619 15:04:58.392405 4390 ts_tablet_manager.cc:1939] T 3164dfb6423d4119a506958b5249f840 P bfbc366cdb934ccda6bd64d44bd7fa80: Deleting consensus metadata
I20260619 15:04:58.393626 3840 catalog_manager.cc:4841] TS 52259d4dca0047f29eed7fcf9cb958fa (127.24.103.67:45181): tablet 3164dfb6423d4119a506958b5249f840 (table test-table [id=18d93d80421c44adb38562ceeb7825f3]) successfully deleted
I20260619 15:04:58.394543 4391 ts_tablet_manager.cc:1918] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:04:58.395059 4391 log.cc:1196] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/3164dfb6423d4119a506958b5249f840
I20260619 15:04:58.395740 4391 ts_tablet_manager.cc:1939] T 3164dfb6423d4119a506958b5249f840 P 1e4cfed7941047e1a811ef3b2f8d5ce5: Deleting consensus metadata
I20260619 15:04:58.395942 3841 catalog_manager.cc:4841] TS bfbc366cdb934ccda6bd64d44bd7fa80 (127.24.103.65:37817): tablet 3164dfb6423d4119a506958b5249f840 (table test-table [id=18d93d80421c44adb38562ceeb7825f3]) successfully deleted
I20260619 15:04:58.398001 3841 catalog_manager.cc:4841] TS 1e4cfed7941047e1a811ef3b2f8d5ce5 (127.24.103.66:46865): tablet 3164dfb6423d4119a506958b5249f840 (table test-table [id=18d93d80421c44adb38562ceeb7825f3]) successfully deleted
Jun 19 15:04:58 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:04:58.411809 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.371374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.24.103.65:37817)
0619 15:04:58.371634 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:58.371644 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:58.391482 (+ 19838us) server_negotiation.cc:407] Connection header received
0619 15:04:58.391831 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:58.391841 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:58.391951 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:58.392044 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:58.394243 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.395633 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.398213 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.398854 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.398965 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.403971 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:58.403987 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:58.404003 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:58.404047 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:58.408133 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:58.408981 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:58.408990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:58.408992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:58.409168 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:58.409611 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:58.409618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:58.409621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:58.410200 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:58.410636 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:58.411069 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 15:04:58.411317 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 15:04:58.412572 4064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:58.515202 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 3936
I20260619 15:04:58.534238 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 4075
Jun 19 15:04:58 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:04:58.550437 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.535181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.24.103.66:46865)
0619 15:04:58.535345 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:58.535350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:58.535368 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:04:58.535522 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:58.535526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:58.535597 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:58.535677 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:58.536970 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.537741 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.539406 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.539904 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.540002 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.544505 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:58.544543 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:58.544560 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:58.544603 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:58.547612 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:58.548389 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:58.548396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:58.548398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:58.548522 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:58.548960 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:58.548963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:58.548966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:58.549353 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:58.549596 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:58.549877 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 15:04:58.550046 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 15:04:58.551252 4207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:58.641079 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 4075
I20260619 15:04:58.660393 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 4218
Jun 19 15:04:58 dist-test-slave-zpbw krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:04:58.676800 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:04:58.661219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34770 (local address 127.24.103.67:45181)
0619 15:04:58.661432 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 15:04:58.661440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:04:58.661467 (+ 27us) server_negotiation.cc:407] Connection header received
0619 15:04:58.661668 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:04:58.661673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:04:58.661757 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:04:58.661863 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:04:58.663261 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.664091 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.665360 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:04:58.665800 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:04:58.665984 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:04:58.670464 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:04:58.670477 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:04:58.670491 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:04:58.670532 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:04:58.673825 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:58.674655 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:58.674662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:58.674665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:58.674831 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:04:58.675241 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:04:58.675247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:04:58.675251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:04:58.675625 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:04:58.675909 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 15:04:58.676227 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 15:04:58.676441 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 15:04:58.677467 4341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:58.765595 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 4218
I20260619 15:04:58.781879 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 3824
I20260619 15:04:58.783175 3886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:04:58.896355 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 3824
I20260619 15:04:58.918061 24989 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 15:04:59.255774 24989 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 15:04:59.256059 24989 mini_ranger.cc:66] Stopping Ranger...
I20260619 15:04:59.602308 24989 mini_ranger.cc:68] Stopped Ranger
2026-06-19 15:04:59.602 UTC [3410] LOG: received smart shutdown request
2026-06-19 15:04:59.605 UTC [3410] LOG: background worker "logical replication launcher" (PID 3419) exited with exit code 1
2026-06-19 15:04:59.606 UTC [3414] LOG: shutting down
2026-06-19 15:04:59.746 UTC [3410] LOG: database system is shut down
2026-06-19T15:04:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49320 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:04:59 dist-test-slave-zpbw krb5kdc[4405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:04:59 dist-test-slave-zpbw krb5kdc[4405](info): set up 2 sockets
Jun 19 15:04:59 dist-test-slave-zpbw krb5kdc[4405](info): commencing operation
krb5kdc: starting...
W20260619 15:05:00.979341 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:05:01 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:05:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:05:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/rangeradmin_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/rangeradmin_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/rangerlookup_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/rangerlookup_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
I20260619 15:05:01.309275 24989 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskn730WE/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/postgres -l logfile start
2026-06-19 15:05:03.738 UTC [4435] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 15:05:03.738 UTC [4435] LOG: listening on IPv4 address "127.24.103.84", port 52619
2026-06-19 15:05:03.742 UTC [4435] LOG: listening on Unix socket "/tmp/.s.PGSQL.52619"
2026-06-19 15:05:03.765 UTC [4438] LOG: database system was shut down at 2026-06-19 15:05:02 UTC
2026-06-19 15:05:03.769 UTC [4435] LOG: database system is ready to accept connections
I20260619 15:05:04.743119 24989 mini_postgres.cc:96] Postgres bound to 52619
2026-06-19 15:05:04.760 UTC [4446] FATAL: database "slave" does not exist
127.24.103.84:52619 - accepting connections
I20260619 15:05:04.760937 24989 mini_ranger.cc:161] Starting Ranger...
I20260619 15:05:04.788221 24989 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 15:05:05.241776 24989 mini_ranger.cc:87] Created ranger Postgres database
I20260619 15:05:05.241873 24989 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-admin
2026-06-19 15:05:05,542 [I] DB FLAVOR :POSTGRES
2026-06-19 15:05:05,543 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 15:05:05,543 [I] Checking connection..
2026-06-19 15:05:05,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 15:05:05,792 [I] Checking connection passed.
2026-06-19 15:05:05,792 [I] --------- Verifying version history table ---------
2026-06-19 15:05:05,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 15:05:06,041 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 15:05:06,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 15:05:06,287 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 15:05:06,288 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 15:05:06,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 15:05:06.525 UTC [4510] WARNING: there is no transaction in progress
2026-06-19 15:05:06,531 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 15:05:06,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 15:05:06,787 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 15:05:06,788 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 15:05:06,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 15:05:07,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 15:05:07,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zpbw.c.gcp-upstream.internal','N') ;"
2026-06-19 15:05:07,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 15:05:07,778 [I] Table x_portal_user does not exist in database ranger
2026-06-19 15:05:07,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 15:05:08,036 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 15:05:08,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 15:05:08,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 15:05:08,531 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 15:05:08,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 15:05:09.469 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.484 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.499 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.514 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.531 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.547 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.600 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.607 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.620 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.630 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.640 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.654 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.664 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.671 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.676 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:09.683 UTC [4618] WARNING: there is no transaction in progress
2026-06-19 15:05:10,289 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 15:05:10,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 15:05:10,533 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 15:05:10,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 15:05:10,801 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 15:05:10,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 15:05:11,057 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 15:05:11,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 15:05:11,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 15:05:11,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zpbw.c.gcp-upstream.internal';"
2026-06-19 15:05:11,784 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 15:05:11,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 15:05:12,044 [I] DB_PATCHES have already been applied
I20260619 15:05:12.050381 24989 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-admin:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskn730WE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 15:05:12.050524 24989 mini_ranger.cc:193] Using host: 127.24.103.84
I20260619 15:05:12.065230 24989 mini_ranger.cc:227] Ranger admin URL: http://127.24.103.84:39835
Jun 19, 2026 3:05:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 3:05:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 3:05:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 3:05:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 3:05:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 3:05:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 3:05:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 15:05:13.225469 24989 mini_ranger.cc:160] Time spent starting Ranger: real 8.465s user 0.004s sys 0.036s
Jun 19, 2026 3:05:33 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 3:05:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 3:05:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 3:05:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 3:05:33 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 3:05:34 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 15:05:35.184525 24989 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/rangerkms_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/rangerkms_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.103.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.103.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 15:05:35 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881535, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 15:05:35.345014 24989 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 15:05:35.366171 24989 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 15:05:36.118484 24989 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 15:05:36.118641 24989 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-kms
I20260619 15:05:36.118728 24989 mini_ranger_kms.cc:227] Using postgres at 127.24.103.84:52619
2026-06-19 15:05:36,469 [I] DB FLAVOR :POSTGRES
2026-06-19 15:05:36,470 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 15:05:36,470 [I] Checking connection
2026-06-19 15:05:36,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 15:05:36,714 [I] connection success
2026-06-19 15:05:36,714 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 15:05:36,714 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 15:05:36,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 15:05:36,968 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 15:05:36,969 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 15:05:36,969 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 15:05:36,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:52619/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 15:05:37,242 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 15:05:37.713958 24989 mini_ranger_kms.cc:310] Created kms service
I20260619 15:05:37.907409 24989 mini_ranger_kms.cc:326] Created kudu user
I20260619 15:05:37.953521 24989 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 15:05:37 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881535, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:05:38.362828 24989 mini_ranger_kms.cc:384] Added ranger policy
I20260619 15:05:38.363168 24989 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-kms:/tmp/dist-test-taskn730WE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskn730WE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 15:05:38.363219 24989 mini_ranger_kms.cc:240] Using host: 127.24.103.84
I20260619 15:05:38.380051 24989 mini_ranger_kms.cc:276] Ranger KMS PID: 4798
I20260619 15:05:38.380192 24989 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.103.84:40811
Jun 19, 2026 3:05:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 3:05:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 3:05:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 3:05:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 3:05:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 3:05:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 15:05:40.811049 24989 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.466s user 0.079s sys 0.290s
I20260619 15:05:40.811303 24989 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 15:05:40.811477 24989 mini_ranger_kms.cc:401] 127.24.103.84:40811/kms/v1/keys
Jun 19 15:05:40 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781881540, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.103.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 15:05:41 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881540, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.103.84@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
Jun 19 15:05:41 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881540, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.103.84@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
Jun 19 15:05:44 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:05:44.154294 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:36547
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:36547
--ranger_config_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:40811/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:05:44.492654 4849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:05:44.492959 4849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:05:44.493046 4849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:05:44.500604 4849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:05:44.500728 4849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:05:44.500774 4849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:05:44.500823 4849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:05:44.500864 4849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:05:44.511302 4849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.103.84:40811/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:36547
--ranger_config_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:05:44.512595 4849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:05:44.513887 4849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:05:44.526101 4857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:44.526219 4854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:44.526489 4849 server_base.cc:1034] running on GCE node
W20260619 15:05:44.526165 4855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:44.527618 4849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:05:44.529027 4849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:05:44.530269 4849 hybrid_clock.cc:648] HybridClock initialized: now 1781881544530213 us; error 49 us; skew 500 ppm
I20260619 15:05:44.530635 4849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:05:44 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881544, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:44.533773 4849 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:05:44.535394 4849 webserver.cc:460] Webserver started at http://127.24.103.126:37497/ using document root <none> and password file <none>
I20260619 15:05:44.536059 4849 fs_manager.cc:359] Metadata directory not provided
I20260619 15:05:44.536156 4849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:05:44.536412 4849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 15:05:44 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881544, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:05:44.593809 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "787e4047bab34150a2c6380a22f461c9"
format_stamp: "Formatted at 2026-06-19 15:05:44 on dist-test-slave-zpbw"
server_key: "0ab2a20346de069c8ebd6b824af26090"
server_key_iv: "581f800d847d4f98d400dc2d06931ec8"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:44.594641 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "787e4047bab34150a2c6380a22f461c9"
format_stamp: "Formatted at 2026-06-19 15:05:44 on dist-test-slave-zpbw"
server_key: "0ab2a20346de069c8ebd6b824af26090"
server_key_iv: "581f800d847d4f98d400dc2d06931ec8"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:44.600193 4849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 15:05:44.620764 4864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:44.622598 4849 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260619 15:05:44.622831 4849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "787e4047bab34150a2c6380a22f461c9"
format_stamp: "Formatted at 2026-06-19 15:05:44 on dist-test-slave-zpbw"
server_key: "0ab2a20346de069c8ebd6b824af26090"
server_key_iv: "581f800d847d4f98d400dc2d06931ec8"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:44.623098 4849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:05:44.645848 4849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:05:44.649750 4849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:05:44.650017 4849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:05:44.669001 4849 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:36547
I20260619 15:05:44.669018 4915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:36547 every 8 connection(s)
I20260619 15:05:44.670456 4849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:05:44.674301 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:05:44.680337 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 4849
I20260619 15:05:44.680578 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:05:44.681483 24989 external_mini_cluster.cc:1428] Setting key 209888296cf42cb6a49741a860d84aba
I20260619 15:05:44.686020 4916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9: Bootstrap starting.
I20260619 15:05:44.690019 4916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9: Neither blocks nor log segments found. Creating new log.
I20260619 15:05:44.691299 4916 log.cc:824] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9: Log is configured to *not* fsync() on all Append() calls
I20260619 15:05:44.694730 4916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9: No bootstrap required, opened a new log
Jun 19 15:05:44 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:05:44.705436 4916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787e4047bab34150a2c6380a22f461c9" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36547 } }
I20260619 15:05:44.705932 4916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:05:44.706038 4916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787e4047bab34150a2c6380a22f461c9, State: Initialized, Role: FOLLOWER
I20260619 15:05:44.706754 4916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787e4047bab34150a2c6380a22f461c9" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36547 } }
I20260619 15:05:44.707083 4916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:05:44.707226 4916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:05:44.707409 4916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:05:44.707433 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:44.683748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.24.103.126:36547)
0619 15:05:44.684647 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:44.684659 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:44.684703 (+ 44us) server_negotiation.cc:407] Connection header received
0619 15:05:44.686563 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:44.686588 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:44.687082 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:44.687425 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:44.689299 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:44.690883 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:44.692626 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:44.693355 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:44.693563 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:44.698370 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:44.698382 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:44.698409 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:44.698460 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:44.702547 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:44.703398 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:44.703408 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:44.703418 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:44.703636 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:44.704027 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:44.704032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:44.704042 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:44.704816 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:44.705108 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:44.705526 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 15:05:44.705968 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":394,"threads_started":1}
I20260619 15:05:44.710479 4916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787e4047bab34150a2c6380a22f461c9" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36547 } }
I20260619 15:05:44.711068 4916 leader_election.cc:302] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 787e4047bab34150a2c6380a22f461c9; no voters:
I20260619 15:05:44.711628 4916 leader_election.cc:288] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:05:44.711779 4921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:05:44.712687 4921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 1 LEADER]: Becoming Leader. State: Replica: 787e4047bab34150a2c6380a22f461c9, State: Running, Role: LEADER
I20260619 15:05:44.713315 4921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787e4047bab34150a2c6380a22f461c9" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36547 } }
I20260619 15:05:44.713851 4916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:05:44.717443 4922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "787e4047bab34150a2c6380a22f461c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787e4047bab34150a2c6380a22f461c9" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36547 } } }
I20260619 15:05:44.718052 4922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [sys.catalog]: This master's current role is: LEADER
I20260619 15:05:44.717453 4923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 787e4047bab34150a2c6380a22f461c9. Latest consensus state: current_term: 1 leader_uuid: "787e4047bab34150a2c6380a22f461c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787e4047bab34150a2c6380a22f461c9" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36547 } } }
I20260619 15:05:44.718609 4923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [sys.catalog]: This master's current role is: LEADER
I20260619 15:05:44.727946 4929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:05:44.728739 4916 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 15:05:44.732712 4929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:05:44.751500 4929 catalog_manager.cc:1347] Generated new cluster ID: 1a8ca65423b342b6aa68047aa129d228
I20260619 15:05:44.751766 4929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:05:44.773422 4929 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:05:44.774958 4929 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:05:44.785241 4929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9: Generated new TSK 0
I20260619 15:05:44.786201 4929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:05:45.572046 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:40811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36547
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 15:05:45 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781881545, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 15:05:45.991724 4961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:05:45.992127 4961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:05:45.992332 4961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:05:46.002514 4961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:05:46.002632 4961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:05:46.002874 4961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:05:46.018258 4961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.103.84:40811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:05:46.019969 4961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:05:46.021647 4961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:05:46.039695 4972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:46.040053 4971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:46.051082 4961 server_base.cc:1034] running on GCE node
W20260619 15:05:46.052107 4974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:46.053023 4961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:05:46.054287 4961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:05:46.055567 4961 hybrid_clock.cc:648] HybridClock initialized: now 1781881546055505 us; error 48 us; skew 500 ppm
I20260619 15:05:46.055958 4961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:05:46 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881546, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:46.068079 4961 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:05:46.079887 4961 webserver.cc:460] Webserver started at http://127.24.103.65:40427/ using document root <none> and password file <none>
I20260619 15:05:46.080772 4961 fs_manager.cc:359] Metadata directory not provided
I20260619 15:05:46.080943 4961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:05:46.081297 4961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 15:05:46 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881546, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:05:46.105307 4961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "43e979620e4e4ee58666a27cb54b7071"
format_stamp: "Formatted at 2026-06-19 15:05:46 on dist-test-slave-zpbw"
server_key: "d7da049133e6587540a7c73f25885838"
server_key_iv: "f4f1712f03d543757c5b6e8c8c709bb2"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:46.106559 4961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "43e979620e4e4ee58666a27cb54b7071"
format_stamp: "Formatted at 2026-06-19 15:05:46 on dist-test-slave-zpbw"
server_key: "d7da049133e6587540a7c73f25885838"
server_key_iv: "f4f1712f03d543757c5b6e8c8c709bb2"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:46.129323 4961 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.018s sys 0.003s
I20260619 15:05:46.159504 4981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:46.161672 4961 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.008s sys 0.002s
I20260619 15:05:46.161989 4961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "43e979620e4e4ee58666a27cb54b7071"
format_stamp: "Formatted at 2026-06-19 15:05:46 on dist-test-slave-zpbw"
server_key: "d7da049133e6587540a7c73f25885838"
server_key_iv: "f4f1712f03d543757c5b6e8c8c709bb2"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:46.162397 4961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:05:46.231519 4961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:05:46.237277 4961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:05:46.237651 4961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:05:46.251307 4961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:05:46.259693 4961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:05:46.259827 4961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:46.259969 4961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:05:46.260031 4961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 15:05:46 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881545, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:05:46.822227 4961 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:35635
I20260619 15:05:46.822273 5093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:35635 every 8 connection(s)
I20260619 15:05:46.823940 4961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:05:46.830852 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 4961
I20260619 15:05:46.831063 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:05:46.831903 24989 external_mini_cluster.cc:1428] Setting key fdf02ebb19cc725f6a8ded150fa27212
Jun 19 15:05:46 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881546, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:05:46.864691 5096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:46.827023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:45569 (local address 127.24.103.126:36547)
0619 15:05:46.835212 (+ 8189us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:46.835218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:46.837820 (+ 2602us) server_negotiation.cc:407] Connection header received
0619 15:05:46.839345 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:46.839352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:46.839458 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:46.839558 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:46.842090 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:46.843332 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:46.849975 (+ 6643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:46.850868 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:46.851000 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:46.856215 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:46.856224 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:46.856229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:46.856277 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:46.860225 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:46.861458 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:46.861466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:46.861470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:46.861651 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:46.862263 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:46.862269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:46.862272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:46.862664 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:46.862812 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:46.863901 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0619 15:05:46.864170 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8046,"thread_start_us":7906,"threads_started":1}
I20260619 15:05:46.867251 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:46.827596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36547 (local address 127.24.103.65:45569)
0619 15:05:46.836158 (+ 8562us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:46.836214 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:46.838859 (+ 2645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:46.840194 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:46.840208 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:46.840935 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:46.841864 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:46.841891 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:46.843580 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:46.843587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:46.848455 (+ 4868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:46.848482 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:46.851458 (+ 2976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:46.851469 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:46.851824 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:46.852683 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:46.852713 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:46.855843 (+ 3130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:46.860488 (+ 4645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:46.860501 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:46.860526 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:46.861287 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:46.861833 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:46.861838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:46.861842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:46.862123 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:46.862918 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:46.862927 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:46.863417 (+ 490us) client_negotiation.cc:769] Sending connection context
0619 15:05:46.865054 (+ 1637us) client_negotiation.cc:240] Negotiation successful
0619 15:05:46.865670 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8162,"thread_start_us":20740,"threads_started":1}
I20260619 15:05:46.869339 5094 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36547
I20260619 15:05:46.869730 5094 heartbeater.cc:464] Registering TS with master...
I20260619 15:05:46.870746 5094 heartbeater.cc:511] Master 127.24.103.126:36547 requested a full tablet report, sending...
I20260619 15:05:46.873623 4881 ts_manager.cc:194] Registered new tserver with Master: 43e979620e4e4ee58666a27cb54b7071 (127.24.103.65:35635)
I20260619 15:05:46.875758 4881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:45569
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Jun 19 15:05:46 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881545, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:05:46.927562 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:40811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36547
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:05:47.373319 5101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:05:47.373656 5101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:05:47.373821 5101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:05:47.381470 5101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:05:47.381631 5101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:05:47.381815 5101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:05:47.392180 5101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.103.84:40811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:05:47.393617 5101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:05:47.395066 5101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:05:47.408812 5115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:47.408812 5114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:47.409613 5117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:47.410071 5101 server_base.cc:1034] running on GCE node
I20260619 15:05:47.410893 5101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:05:47.412143 5101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:05:47.413400 5101 hybrid_clock.cc:648] HybridClock initialized: now 1781881547413323 us; error 68 us; skew 500 ppm
I20260619 15:05:47.413784 5101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:05:47 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881547, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:47.417744 5101 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:05:47.419550 5101 webserver.cc:460] Webserver started at http://127.24.103.66:37375/ using document root <none> and password file <none>
I20260619 15:05:47.420284 5101 fs_manager.cc:359] Metadata directory not provided
I20260619 15:05:47.420411 5101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:05:47.420740 5101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 15:05:47 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881547, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:05:47.439414 5101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "4c85b32c330a463b846c873af03eae28"
format_stamp: "Formatted at 2026-06-19 15:05:47 on dist-test-slave-zpbw"
server_key: "d72265d99a16150f99e645e53f4dd296"
server_key_iv: "367431da97a7ee0ac7060b9baa7fc65e"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:47.440593 5101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "4c85b32c330a463b846c873af03eae28"
format_stamp: "Formatted at 2026-06-19 15:05:47 on dist-test-slave-zpbw"
server_key: "d72265d99a16150f99e645e53f4dd296"
server_key_iv: "367431da97a7ee0ac7060b9baa7fc65e"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:47.447849 5101 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260619 15:05:47.465829 5124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:47.467489 5101 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260619 15:05:47.467741 5101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "4c85b32c330a463b846c873af03eae28"
format_stamp: "Formatted at 2026-06-19 15:05:47 on dist-test-slave-zpbw"
server_key: "d72265d99a16150f99e645e53f4dd296"
server_key_iv: "367431da97a7ee0ac7060b9baa7fc65e"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:47.468031 5101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:05:47.481977 5101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:05:47.486299 5101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:05:47.486629 5101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:05:47.487725 5101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:05:47.489279 5101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:05:47.489379 5101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:47.489539 5101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:05:47.489611 5101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:47.528321 5101 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:45423
I20260619 15:05:47.528302 5236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:45423 every 8 connection(s)
I20260619 15:05:47.529690 5101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:05:47.539361 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 5101
I20260619 15:05:47.539577 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:05:47.540127 24989 external_mini_cluster.cc:1428] Setting key fd084ff3b03c3f25b3cc6fcf1567f8bc
Jun 19 15:05:47 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881547, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:05:47.554764 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:47.532553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:40537 (local address 127.24.103.126:36547)
0619 15:05:47.533062 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:47.533066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:47.534150 (+ 1084us) server_negotiation.cc:407] Connection header received
0619 15:05:47.536078 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:47.536083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:47.536172 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:47.536319 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:47.537896 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:47.538668 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:47.540552 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:47.541152 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:47.541296 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:47.546559 (+ 5263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:47.546567 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:47.546571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:47.546617 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:47.550288 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:47.551502 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:47.551517 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:47.551521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:47.551680 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:47.552271 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:47.552276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:47.552279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:47.552650 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:47.552837 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:47.553938 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0619 15:05:47.554229 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":272,"threads_started":1}
I20260619 15:05:47.555914 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:47.533224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36547 (local address 127.24.103.66:40537)
0619 15:05:47.533986 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:47.534025 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:47.535772 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:47.536592 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:47.536620 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:47.537073 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:47.537712 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:47.537731 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:47.538875 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:47.538879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:47.540406 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:47.540420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:47.541389 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:47.541398 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:47.541664 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:47.542992 (+ 1328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:47.543026 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:47.546211 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:47.550511 (+ 4300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:47.550524 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:47.550555 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:47.551327 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:47.551812 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:47.551819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:47.551826 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:47.552123 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:47.552918 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:47.552928 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:47.553415 (+ 487us) client_negotiation.cc:769] Sending connection context
0619 15:05:47.553781 (+ 366us) client_negotiation.cc:240] Negotiation successful
0619 15:05:47.554369 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":327,"threads_started":1}
I20260619 15:05:47.557708 5237 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36547
I20260619 15:05:47.558063 5237 heartbeater.cc:464] Registering TS with master...
I20260619 15:05:47.559031 5237 heartbeater.cc:511] Master 127.24.103.126:36547 requested a full tablet report, sending...
I20260619 15:05:47.560897 4881 ts_manager.cc:194] Registered new tserver with Master: 4c85b32c330a463b846c873af03eae28 (127.24.103.66:45423)
I20260619 15:05:47.562052 4881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:40537
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:05:47.594097 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:40811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36547
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:05:47.881727 5094 heartbeater.cc:503] Master 127.24.103.126:36547 was elected leader, sending a full tablet report...
W20260619 15:05:47.942723 5243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:05:47.943210 5243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:05:47.943425 5243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:05:47.950979 5243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:05:47.951104 5243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:05:47.951367 5243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:05:47.961606 5243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.103.84:40811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:05:47.963114 5243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:05:47.964496 5243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:05:47.978873 5248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:47.978830 5251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:47.978917 5249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:47.984362 5243 server_base.cc:1034] running on GCE node
I20260619 15:05:47.985059 5243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:05:47.986068 5243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:05:47.987293 5243 hybrid_clock.cc:648] HybridClock initialized: now 1781881547987257 us; error 59 us; skew 500 ppm
I20260619 15:05:47.987618 5243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:05:47 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881547, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:47.990845 5243 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:05:47.992426 5243 webserver.cc:460] Webserver started at http://127.24.103.67:35645/ using document root <none> and password file <none>
I20260619 15:05:47.993170 5243 fs_manager.cc:359] Metadata directory not provided
I20260619 15:05:47.993315 5243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:05:47.993652 5243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 15:05:47 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881547, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:05:48.007584 5243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "23daedc0b2224346a687872aa008eac0"
format_stamp: "Formatted at 2026-06-19 15:05:48 on dist-test-slave-zpbw"
server_key: "e836c55c32c284a90e0adbbfe37aa109"
server_key_iv: "00d843c5341d8d3e3aa95e701d468b37"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:48.008431 5243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "23daedc0b2224346a687872aa008eac0"
format_stamp: "Formatted at 2026-06-19 15:05:48 on dist-test-slave-zpbw"
server_key: "e836c55c32c284a90e0adbbfe37aa109"
server_key_iv: "00d843c5341d8d3e3aa95e701d468b37"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:48.014263 5243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:05:48.031689 5258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:48.033463 5243 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 15:05:48.033706 5243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "23daedc0b2224346a687872aa008eac0"
format_stamp: "Formatted at 2026-06-19 15:05:48 on dist-test-slave-zpbw"
server_key: "e836c55c32c284a90e0adbbfe37aa109"
server_key_iv: "00d843c5341d8d3e3aa95e701d468b37"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:48.033987 5243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:05:48.050716 5243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:05:48.054829 5243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:05:48.055172 5243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:05:48.056249 5243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:05:48.057662 5243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:05:48.057735 5243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:48.057875 5243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:05:48.057927 5243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:48.091033 5243 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:44771
I20260619 15:05:48.091064 5370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:44771 every 8 connection(s)
I20260619 15:05:48.092523 5243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:05:48.099180 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 5243
I20260619 15:05:48.099375 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:05:48.099843 24989 external_mini_cluster.cc:1428] Setting key c21cef7618e8ae832420f195c9508b23
Jun 19 15:05:48 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881547, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:05:48.112679 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:48.095403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:56161 (local address 127.24.103.126:36547)
0619 15:05:48.095885 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:48.095890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:48.096798 (+ 908us) server_negotiation.cc:407] Connection header received
0619 15:05:48.098564 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:48.098570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:48.098667 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:48.098824 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:48.100443 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:48.101331 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:48.102976 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:48.103530 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:48.103665 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:48.106973 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:48.106979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:48.106982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:48.107022 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:48.109381 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:48.110199 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:48.110203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:48.110205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:48.110331 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:48.110823 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:48.110834 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:48.110837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:48.111131 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:48.111328 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:48.112065 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 15:05:48.112304 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":257,"threads_started":1}
I20260619 15:05:48.113639 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:48.095923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36547 (local address 127.24.103.67:56161)
0619 15:05:48.096634 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:48.096672 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:48.098258 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:48.099064 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:48.099073 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:48.099621 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:48.100252 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:48.100269 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:48.101490 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:48.101494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:48.102766 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:48.102835 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:48.103753 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:48.103760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:48.104003 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:48.104625 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:48.104647 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:48.106695 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:48.109555 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:48.109565 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:48.109585 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:48.110052 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:48.110449 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:48.110453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:48.110457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:48.110677 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:48.111293 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:48.111302 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:48.111684 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 15:05:48.111962 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 15:05:48.112517 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":275,"threads_started":1}
I20260619 15:05:48.115254 5371 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36547
I20260619 15:05:48.115592 5371 heartbeater.cc:464] Registering TS with master...
I20260619 15:05:48.116292 5371 heartbeater.cc:511] Master 127.24.103.126:36547 requested a full tablet report, sending...
I20260619 15:05:48.117627 4881 ts_manager.cc:194] Registered new tserver with Master: 23daedc0b2224346a687872aa008eac0 (127.24.103.67:44771)
I20260619 15:05:48.118479 4881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:56161
I20260619 15:05:48.128422 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:05:48.130285 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 4961
Jun 19 15:05:48 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:05:48.147750 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:48.131427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.24.103.65:35635)
0619 15:05:48.132125 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:48.132133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:48.132160 (+ 27us) server_negotiation.cc:407] Connection header received
0619 15:05:48.132217 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:48.132223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:48.132442 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:48.132579 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:48.134160 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:48.135352 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:48.136705 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:48.137248 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:48.137343 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:48.141749 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:48.141762 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:48.141778 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:48.141815 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:48.144797 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:48.145678 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:48.145685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:48.145687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:48.145822 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:48.146156 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:48.146159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:48.146161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:48.146514 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:48.146756 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:48.147121 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 15:05:48.147341 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":368,"threads_started":1}
I20260619 15:05:48.149107 5089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:05:48.232218 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 4961
I20260619 15:05:48.247991 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 5101
Jun 19 15:05:48 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:05:48.264763 5378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:48.248825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33770 (local address 127.24.103.66:45423)
0619 15:05:48.249356 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:48.249363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:48.249389 (+ 26us) server_negotiation.cc:407] Connection header received
0619 15:05:48.249449 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:48.249455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:48.249727 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:48.249874 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:48.251141 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:48.252089 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:48.253472 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:48.254015 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:48.254111 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:48.258953 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:48.258978 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:48.258994 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:48.259035 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:48.262088 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:48.262907 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:48.262915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:48.262917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:48.263065 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:48.263403 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:48.263406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:48.263408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:48.263754 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:48.263986 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:48.264253 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 15:05:48.264455 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":277,"threads_started":1}
I20260619 15:05:48.266024 5232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:05:48.349190 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 5101
I20260619 15:05:48.365561 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 5243
Jun 19 15:05:48 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:05:48.382109 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:48.366640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.24.103.67:44771)
0619 15:05:48.367342 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:48.367349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:48.367377 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:05:48.367434 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:48.367441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:48.367675 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:48.367798 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:48.369044 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:48.370025 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:48.371450 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:48.372019 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:48.372115 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:48.376623 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:48.376636 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:48.376651 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:48.376689 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:48.379605 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:48.380229 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:48.380236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:48.380238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:48.380362 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:48.380693 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:48.380696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:48.380698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:48.381029 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:48.381263 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:48.381501 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 15:05:48.381703 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":413,"threads_started":1}
I20260619 15:05:48.383549 5366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:05:48.468274 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 5243
I20260619 15:05:48.484123 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 4849
I20260619 15:05:48.485412 4911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:05:48.592530 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 4849
I20260619 15:05:48.616624 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:36547
--webserver_interface=127.24.103.126
--webserver_port=37497
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:36547
--ranger_config_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:40811/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:40811/kms with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:05:48.945427 5383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:05:48.945799 5383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:05:48.945971 5383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:05:48.953424 5383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:05:48.953603 5383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:05:48.953732 5383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:05:48.953828 5383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:05:48.953920 5383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:05:48.964349 5383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:40811/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:36547
--ranger_config_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=37497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:05:48.965775 5383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:05:48.967185 5383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:05:48.979650 5388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:48.979650 5389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:48.979671 5391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:48.980342 5383 server_base.cc:1034] running on GCE node
I20260619 15:05:48.981180 5383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:05:48.982636 5383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:05:48.983886 5383 hybrid_clock.cc:648] HybridClock initialized: now 1781881548983860 us; error 67 us; skew 500 ppm
I20260619 15:05:48.984244 5383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:05:48 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881548, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:48.987228 5383 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:05:48.988812 5383 webserver.cc:460] Webserver started at http://127.24.103.126:37497/ using document root <none> and password file <none>
I20260619 15:05:48.989476 5383 fs_manager.cc:359] Metadata directory not provided
I20260619 15:05:48.989651 5383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 15:05:49 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881548, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:05:49.041940 5383 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 15:05:49.045125 5399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:49.046612 5383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:05:49.046880 5383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "787e4047bab34150a2c6380a22f461c9"
format_stamp: "Formatted at 2026-06-19 15:05:44 on dist-test-slave-zpbw"
server_key: "0ab2a20346de069c8ebd6b824af26090"
server_key_iv: "581f800d847d4f98d400dc2d06931ec8"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:49.047894 5383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:05:49.072265 5383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:05:49.076148 5383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:05:49.076453 5383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:05:49.095490 5383 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:36547
I20260619 15:05:49.095515 5450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:36547 every 8 connection(s)
I20260619 15:05:49.097056 5383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:05:49.100736 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 5383
I20260619 15:05:49.101599 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:40811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:35635
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:36547
--webserver_port=40427
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:40811/kms with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:05:49.104223 5451 sys_catalog.cc:263] Verifying existing consensus state
I20260619 15:05:49.107643 5451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9: Bootstrap starting.
I20260619 15:05:49.126962 5451 log.cc:824] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9: Log is configured to *not* fsync() on all Append() calls
I20260619 15:05:49.136071 5451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:05:49.136643 5451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9: Bootstrap complete.
I20260619 15:05:49.145712 5451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787e4047bab34150a2c6380a22f461c9" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36547 } }
I20260619 15:05:49.146632 5451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787e4047bab34150a2c6380a22f461c9, State: Initialized, Role: FOLLOWER
I20260619 15:05:49.147310 5451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787e4047bab34150a2c6380a22f461c9" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36547 } }
I20260619 15:05:49.147573 5451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:05:49.147713 5451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:05:49.147897 5451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 1 FOLLOWER]: Advancing to term 2
I20260619 15:05:49.152031 5451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787e4047bab34150a2c6380a22f461c9" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36547 } }
I20260619 15:05:49.152630 5451 leader_election.cc:302] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 787e4047bab34150a2c6380a22f461c9; no voters:
I20260619 15:05:49.153343 5451 leader_election.cc:288] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 15:05:49.153483 5455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 15:05:49.155944 5451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:05:49.160454 5455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [term 2 LEADER]: Becoming Leader. State: Replica: 787e4047bab34150a2c6380a22f461c9, State: Running, Role: LEADER
I20260619 15:05:49.161192 5455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787e4047bab34150a2c6380a22f461c9" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36547 } }
I20260619 15:05:49.164542 5456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "787e4047bab34150a2c6380a22f461c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787e4047bab34150a2c6380a22f461c9" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36547 } } }
I20260619 15:05:49.164954 5456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [sys.catalog]: This master's current role is: LEADER
I20260619 15:05:49.165042 5457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 787e4047bab34150a2c6380a22f461c9. Latest consensus state: current_term: 2 leader_uuid: "787e4047bab34150a2c6380a22f461c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787e4047bab34150a2c6380a22f461c9" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36547 } } }
I20260619 15:05:49.165329 5457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9 [sys.catalog]: This master's current role is: LEADER
I20260619 15:05:49.164803 5451 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 15:05:49.166186 5470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:05:49.178169 5470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:05:49.180146 5470 catalog_manager.cc:1259] Loaded cluster ID: 1a8ca65423b342b6aa68047aa129d228
I20260619 15:05:49.180392 5470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:05:49.184908 5470 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:05:49.186882 5470 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9: Loaded TSK: 0
I20260619 15:05:49.188866 5470 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 15:05:49.552682 5453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:05:49.553138 5453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:05:49.553313 5453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:05:49.565560 5453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:05:49.565738 5453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:05:49.565963 5453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:05:49.582284 5453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:40811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:35635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=40427
--tserver_master_addrs=127.24.103.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:05:49.584197 5453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:05:49.586072 5453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:05:49.625243 5488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:49.627650 5489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:49.627707 5491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:49.629753 5453 server_base.cc:1034] running on GCE node
I20260619 15:05:49.630574 5453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:05:49.631946 5453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:05:49.635308 5453 hybrid_clock.cc:648] HybridClock initialized: now 1781881549634732 us; error 560 us; skew 500 ppm
I20260619 15:05:49.635721 5453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:05:49 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881549, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:49.640148 5453 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:05:49.642345 5453 webserver.cc:460] Webserver started at http://127.24.103.65:40427/ using document root <none> and password file <none>
I20260619 15:05:49.643399 5453 fs_manager.cc:359] Metadata directory not provided
I20260619 15:05:49.643573 5453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 15:05:49 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881549, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:05:49.688767 5453 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.009s sys 0.001s
I20260619 15:05:49.693239 5499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:49.695318 5453 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 15:05:49.695536 5453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "43e979620e4e4ee58666a27cb54b7071"
format_stamp: "Formatted at 2026-06-19 15:05:46 on dist-test-slave-zpbw"
server_key: "d7da049133e6587540a7c73f25885838"
server_key_iv: "f4f1712f03d543757c5b6e8c8c709bb2"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:49.697067 5453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:05:49.724633 5453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:05:49.730527 5453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:05:49.730958 5453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:05:49.732424 5453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:05:49.734444 5453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:05:49.734581 5453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:49.734699 5453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:05:49.734822 5453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:49.800379 5453 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:35635
I20260619 15:05:49.800392 5612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:35635 every 8 connection(s)
I20260619 15:05:49.802264 5453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:05:49.804054 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 5453
I20260619 15:05:49.804713 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:40811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:45423
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:36547
--webserver_port=37375
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:40811/kms with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 15:05:49 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881549, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:05:49.869767 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:49.811233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36547 (local address 127.24.103.65:58651)
0619 15:05:49.823480 (+ 12247us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:49.823538 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:49.827522 (+ 3984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:49.830034 (+ 2512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:49.830046 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:49.830820 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:49.831692 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:49.831719 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:49.840826 (+ 9107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:49.840833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:49.843002 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:49.843023 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:49.844655 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:49.844664 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:49.845003 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:49.845888 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:49.845931 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:49.851035 (+ 5104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:49.864439 (+ 13404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:49.864449 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:49.864464 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:49.865150 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:49.865805 (+ 655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:49.865810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:49.865813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:49.866093 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:49.867160 (+ 1067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:49.867170 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:49.867590 (+ 420us) client_negotiation.cc:769] Sending connection context
0619 15:05:49.867863 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 15:05:49.868469 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11914,"thread_start_us":11621,"threads_started":1}
I20260619 15:05:49.873021 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:49.812419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:58651 (local address 127.24.103.126:36547)
0619 15:05:49.827332 (+ 14913us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:49.827353 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:49.827413 (+ 60us) server_negotiation.cc:407] Connection header received
0619 15:05:49.828417 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:49.828440 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:49.828974 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:49.829468 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:49.831973 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:49.840513 (+ 8540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:49.843240 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:49.844356 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:49.844551 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:49.851604 (+ 7053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:49.851620 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:49.851647 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:49.851705 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:49.864169 (+ 12464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:49.865399 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:49.865408 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:49.865416 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:49.865658 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:49.866267 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:49.866272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:49.866275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:49.866979 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:49.870665 (+ 3686us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:49.870988 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 15:05:49.871495 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14637,"thread_start_us":14400,"threads_started":1}
I20260619 15:05:49.876725 5613 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36547
I20260619 15:05:49.877151 5613 heartbeater.cc:464] Registering TS with master...
I20260619 15:05:49.878120 5613 heartbeater.cc:511] Master 127.24.103.126:36547 requested a full tablet report, sending...
W20260619 15:05:49.882661 5613 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:36547 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 15:05:49.883036 5613 heartbeater.cc:464] Registering TS with master...
I20260619 15:05:49.883447 5613 heartbeater.cc:511] Master 127.24.103.126:36547 requested a full tablet report, sending...
I20260619 15:05:49.885277 5613 heartbeater.cc:464] Registering TS with master...
I20260619 15:05:49.885766 5613 heartbeater.cc:511] Master 127.24.103.126:36547 requested a full tablet report, sending...
W20260619 15:05:49.887166 5613 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 15:05:50.262643 5615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:05:50.263103 5615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:05:50.263284 5615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:05:50.275599 5615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:05:50.275780 5615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:05:50.276012 5615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:05:50.292418 5615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:40811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:45423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=37375
--tserver_master_addrs=127.24.103.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:05:50.294206 5615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:05:50.296818 5615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:05:50.314615 5631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:50.315757 5632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:50.317214 5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:50.317814 5615 server_base.cc:1034] running on GCE node
I20260619 15:05:50.318538 5615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:05:50.321571 5615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:05:50.322871 5615 hybrid_clock.cc:648] HybridClock initialized: now 1781881550322793 us; error 68 us; skew 500 ppm
I20260619 15:05:50.323231 5615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:05:50 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881550, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:50.327483 5615 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:05:50.329645 5615 webserver.cc:460] Webserver started at http://127.24.103.66:37375/ using document root <none> and password file <none>
I20260619 15:05:50.330727 5615 fs_manager.cc:359] Metadata directory not provided
I20260619 15:05:50.330956 5615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 15:05:50 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881550, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:05:50.377043 5615 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.000s sys 0.019s
I20260619 15:05:50.390074 5642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:50.392279 5615 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.006s sys 0.005s
I20260619 15:05:50.392592 5615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "4c85b32c330a463b846c873af03eae28"
format_stamp: "Formatted at 2026-06-19 15:05:47 on dist-test-slave-zpbw"
server_key: "d72265d99a16150f99e645e53f4dd296"
server_key_iv: "367431da97a7ee0ac7060b9baa7fc65e"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:50.394304 5615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:05:50.431802 5615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:05:50.437386 5615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:05:50.437705 5615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:05:50.439030 5615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:05:50.440838 5615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:05:50.440956 5615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:50.441107 5615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:05:50.441181 5615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 15:05:50 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781881550, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:50.568706 5615 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:45423
I20260619 15:05:50.568801 5756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:45423 every 8 connection(s)
I20260619 15:05:50.570294 5615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:05:50.574379 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 5615
I20260619 15:05:50.575258 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:40811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:44771
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:36547
--webserver_port=35645
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:40811/kms with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 15:05:50 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881550, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:05:50.623042 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:50.575168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:33363 (local address 127.24.103.126:36547)
0619 15:05:50.582223 (+ 7055us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:50.582234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:50.586500 (+ 4266us) server_negotiation.cc:407] Connection header received
0619 15:05:50.586592 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:50.586599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:50.586703 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:50.595133 (+ 8430us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:50.598200 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:50.599663 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:50.602995 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:50.603922 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:50.604068 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:50.609456 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:50.609464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:50.609470 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:50.609517 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:50.616245 (+ 6728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:50.618477 (+ 2232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:50.618485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:50.618489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:50.618663 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:50.620566 (+ 1903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:50.620572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:50.620579 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:50.621091 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:50.621240 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:50.622297 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0619 15:05:50.622566 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6859,"thread_start_us":6725,"threads_started":1}
I20260619 15:05:50.625672 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:50.574653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36547 (local address 127.24.103.66:33363)
0619 15:05:50.582760 (+ 8107us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:50.582860 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:50.585971 (+ 3111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:50.595816 (+ 9845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:50.595838 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:50.596728 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:50.597847 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:50.597897 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:50.599963 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:50.599971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:50.602693 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:50.602717 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:50.604502 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:50.604513 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:50.604946 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:50.605998 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:50.606040 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:50.609109 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:50.617520 (+ 8411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:50.617533 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:50.617554 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:50.618293 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:50.620006 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:50.620013 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:50.620017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:50.620404 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:50.621356 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:50.621365 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:50.621811 (+ 446us) client_negotiation.cc:769] Sending connection context
0619 15:05:50.623591 (+ 1780us) client_negotiation.cc:240] Negotiation successful
0619 15:05:50.624196 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7736,"thread_start_us":7457,"threads_started":1}
I20260619 15:05:50.627723 5757 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36547
I20260619 15:05:50.628095 5757 heartbeater.cc:464] Registering TS with master...
I20260619 15:05:50.628871 5757 heartbeater.cc:511] Master 127.24.103.126:36547 requested a full tablet report, sending...
I20260619 15:05:50.630854 5416 ts_manager.cc:194] Registered new tserver with Master: 4c85b32c330a463b846c873af03eae28 (127.24.103.66:45423)
I20260619 15:05:50.632583 5416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:33363
I20260619 15:05:50.891412 5613 heartbeater.cc:464] Registering TS with master...
I20260619 15:05:50.891875 5613 heartbeater.cc:511] Master 127.24.103.126:36547 requested a full tablet report, sending...
I20260619 15:05:50.893307 5416 ts_manager.cc:194] Registered new tserver with Master: 43e979620e4e4ee58666a27cb54b7071 (127.24.103.65:35635)
I20260619 15:05:50.894531 5416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:58651
Jun 19 15:05:51 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881550, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
W20260619 15:05:51.104220 5762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:05:51.104811 5762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:05:51.105036 5762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:05:51.118381 5762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:05:51.118662 5762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:05:51.118942 5762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:05:51.137284 5762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:40811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:44771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=35645
--tserver_master_addrs=127.24.103.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:05:51.139417 5762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:05:51.141356 5762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:05:51.180992 5768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:51.181201 5771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:51.181562 5762 server_base.cc:1034] running on GCE node
W20260619 15:05:51.182236 5767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:51.183560 5762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:05:51.189133 5762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:05:51.190693 5762 hybrid_clock.cc:648] HybridClock initialized: now 1781881551190566 us; error 119 us; skew 500 ppm
I20260619 15:05:51.191171 5762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:05:51 dist-test-slave-zpbw krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881551, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:51.201077 5762 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:05:51.229171 5762 webserver.cc:460] Webserver started at http://127.24.103.67:35645/ using document root <none> and password file <none>
I20260619 15:05:51.230440 5762 fs_manager.cc:359] Metadata directory not provided
I20260619 15:05:51.230752 5762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 15:05:51 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881551, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:05:51.270119 5762 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.004s
I20260619 15:05:51.280308 5778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:51.282549 5762 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.006s
I20260619 15:05:51.282891 5762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "23daedc0b2224346a687872aa008eac0"
format_stamp: "Formatted at 2026-06-19 15:05:48 on dist-test-slave-zpbw"
server_key: "e836c55c32c284a90e0adbbfe37aa109"
server_key_iv: "00d843c5341d8d3e3aa95e701d468b37"
server_key_version: "kuduclusterkey@0"
I20260619 15:05:51.284575 5762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:05:51.329100 5762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:05:51.335114 5762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:05:51.335543 5762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:05:51.341425 5762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:05:51.343489 5762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:05:51.343641 5762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:51.343801 5762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:05:51.343915 5762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 15:05:51 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881550, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260619 15:05:51.470471 5762 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:44771
I20260619 15:05:51.471112 5891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:44771 every 8 connection(s)
I20260619 15:05:51.472425 5762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:05:51.477957 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 5762
Jun 19 15:05:51 dist-test-slave-zpbw krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881551, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:05:51.542063 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.482605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36547 (local address 127.24.103.67:42147)
0619 15:05:51.509311 (+ 26706us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:51.509366 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:51.512856 (+ 3490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:51.515732 (+ 2876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:51.515746 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:51.516404 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:51.517253 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:51.517283 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.519930 (+ 2647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.519936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:51.522006 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:51.522024 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.523496 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.523504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:51.523837 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.524694 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:51.524725 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:51.530752 (+ 6027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:51.536963 (+ 6211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:51.536976 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:51.537000 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:51.537781 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:51.538252 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:51.538257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:51.538261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:51.538537 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:51.539309 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:51.539319 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:51.539780 (+ 461us) client_negotiation.cc:769] Sending connection context
0619 15:05:51.540089 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 15:05:51.540661 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24046,"thread_start_us":24427,"threads_started":1}
I20260619 15:05:51.543284 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.483095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:42147 (local address 127.24.103.126:36547)
0619 15:05:51.514135 (+ 31040us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:51.514141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:51.514163 (+ 22us) server_negotiation.cc:407] Connection header received
0619 15:05:51.514230 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:51.514235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:51.514340 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:51.514454 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:51.518106 (+ 3652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.519410 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.522225 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.522900 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.523034 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.531141 (+ 8107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:51.531149 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:51.531154 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:51.531249 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:51.536754 (+ 5505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:51.537942 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:51.537948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:51.537952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:51.538110 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:51.538694 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:51.538700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:51.538704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:51.539150 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:51.539308 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:51.542634 (+ 3326us) server_negotiation.cc:299] Negotiation successful
0619 15:05:51.542905 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30911,"thread_start_us":30462,"threads_started":1}
I20260619 15:05:51.544663 5892 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36547
I20260619 15:05:51.545037 5892 heartbeater.cc:464] Registering TS with master...
I20260619 15:05:51.546067 5892 heartbeater.cc:511] Master 127.24.103.126:36547 requested a full tablet report, sending...
I20260619 15:05:51.547968 5416 ts_manager.cc:194] Registered new tserver with Master: 23daedc0b2224346a687872aa008eac0 (127.24.103.67:44771)
I20260619 15:05:51.548120 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.479132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.24.103.126:36547)
0619 15:05:51.509037 (+ 29905us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:51.509044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:51.515170 (+ 6126us) server_negotiation.cc:407] Connection header received
0619 15:05:51.517513 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:51.517519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:51.517611 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:51.517687 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:51.523414 (+ 5727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.524809 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.526885 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.527693 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.527847 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.531701 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:51.531713 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:51.531719 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:51.531787 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:51.535995 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:51.537984 (+ 1989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:51.537990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:51.537994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:51.538200 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:51.539388 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:51.539393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:51.539397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:51.539748 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:51.540040 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:51.547324 (+ 7284us) server_negotiation.cc:299] Negotiation successful
0619 15:05:51.547651 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29747,"thread_start_us":34687,"threads_started":1}
I20260619 15:05:51.549151 5416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:42147
I20260619 15:05:51.552122 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:05:51.589811 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.556686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.24.103.126:36547)
0619 15:05:51.568581 (+ 11895us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:51.568588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:51.568613 (+ 25us) server_negotiation.cc:407] Connection header received
0619 15:05:51.568687 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:51.568694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:51.568782 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:51.568923 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:51.570885 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.572084 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.577560 (+ 5476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.578233 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.578374 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.582324 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:51.582331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:51.582335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:51.582380 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:51.586356 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:51.587454 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:51.587460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:51.587464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:51.587626 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:51.588168 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:51.588173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:51.588176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:51.588528 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:51.588805 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:51.589220 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 15:05:51.589469 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11716}
I20260619 15:05:51.596858 5416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:05:51.607795 5416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:05:51.637629 5757 heartbeater.cc:503] Master 127.24.103.126:36547 was elected leader, sending a full tablet report...
I20260619 15:05:51.650592 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.637549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:45423 (local address 127.0.0.1:33782)
0619 15:05:51.638294 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:51.638341 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:51.638508 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:51.642227 (+ 3719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:51.642235 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:51.642267 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:05:51.642735 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:51.642752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.644903 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.644908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:51.647293 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:51.647310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.649519 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.649527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:51.649787 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.649851 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 15:05:51.649993 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 15:05:51.650215 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":329,"threads_started":1}
I20260619 15:05:51.652243 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.638260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.24.103.66:45423)
0619 15:05:51.641583 (+ 3323us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:51.641592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:51.641620 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:05:51.641711 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:51.641720 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:51.641981 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:51.643011 (+ 1030us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:05:51.643239 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.644746 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.647537 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.649211 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.651226 (+ 2015us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.651496 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:51.651650 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 15:05:51.651837 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3164,"thread_start_us":3293,"threads_started":1}
I20260619 15:05:51.663195 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.639497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35635 (local address 127.0.0.1:48450)
0619 15:05:51.640639 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:51.640665 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:51.640814 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:51.654641 (+ 13827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:51.654647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:51.654679 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:05:51.655132 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:51.655145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.658114 (+ 2969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.658121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:51.660371 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:51.660390 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.662387 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.662394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:51.662559 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.662598 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 15:05:51.662670 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 15:05:51.662831 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"thread_start_us":767,"threads_started":1}
I20260619 15:05:51.664770 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.640248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.24.103.65:35635)
0619 15:05:51.650405 (+ 10157us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:51.650420 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:51.650472 (+ 52us) server_negotiation.cc:407] Connection header received
0619 15:05:51.650627 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:51.650654 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:51.652041 (+ 1387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:51.654667 (+ 2626us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:05:51.655442 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.657882 (+ 2440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.660587 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.662112 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.663820 (+ 1708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.664084 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:51.664160 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 15:05:51.664392 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":976,"thread_start_us":789,"threads_started":1}
I20260619 15:05:51.669076 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3474b2967843839bc30d54be5a80b6 (DEFAULT_TABLE table=test-table [id=54182d0535db43ed9d34d843026a376c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:05:51.670064 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3474b2967843839bc30d54be5a80b6 (DEFAULT_TABLE table=test-table [id=54182d0535db43ed9d34d843026a376c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:05:51.670742 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3474b2967843839bc30d54be5a80b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:05:51.671510 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3474b2967843839bc30d54be5a80b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:05:51.673267 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.641419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:44771 (local address 127.0.0.1:34506)
0619 15:05:51.650918 (+ 9499us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:51.650940 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:51.651053 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:51.665553 (+ 14500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:51.665561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:51.665598 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:05:51.665787 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:51.665803 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.667901 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.667907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:51.669701 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:51.669729 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.672294 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.672334 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:51.672637 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.672696 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 15:05:51.672789 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 15:05:51.672954 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9383,"thread_start_us":24611,"threads_started":1}
I20260619 15:05:51.674582 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.641752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.24.103.67:44771)
0619 15:05:51.664193 (+ 22441us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:51.664207 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:51.664255 (+ 48us) server_negotiation.cc:407] Connection header received
0619 15:05:51.664390 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:51.664403 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:51.664771 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:51.665546 (+ 775us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:05:51.666020 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.667719 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.669952 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.671828 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.673705 (+ 1877us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.673961 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:51.674047 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 15:05:51.674223 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22162,"thread_start_us":23300,"threads_started":1}
I20260619 15:05:51.680006 5827 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3474b2967843839bc30d54be5a80b6 (DEFAULT_TABLE table=test-table [id=54182d0535db43ed9d34d843026a376c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:05:51.681871 5827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3474b2967843839bc30d54be5a80b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:05:51.695690 5920 tablet_bootstrap.cc:492] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071: Bootstrap starting.
I20260619 15:05:51.701195 5921 tablet_bootstrap.cc:492] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28: Bootstrap starting.
I20260619 15:05:51.702546 5920 tablet_bootstrap.cc:654] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071: Neither blocks nor log segments found. Creating new log.
I20260619 15:05:51.704581 5920 log.cc:824] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071: Log is configured to *not* fsync() on all Append() calls
I20260619 15:05:51.705003 5921 tablet_bootstrap.cc:654] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28: Neither blocks nor log segments found. Creating new log.
I20260619 15:05:51.706076 5922 tablet_bootstrap.cc:492] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0: Bootstrap starting.
I20260619 15:05:51.706223 5921 log.cc:824] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28: Log is configured to *not* fsync() on all Append() calls
I20260619 15:05:51.710192 5922 tablet_bootstrap.cc:654] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0: Neither blocks nor log segments found. Creating new log.
I20260619 15:05:51.710978 5921 tablet_bootstrap.cc:492] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28: No bootstrap required, opened a new log
I20260619 15:05:51.711251 5921 ts_tablet_manager.cc:1397] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260619 15:05:51.711520 5922 log.cc:824] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0: Log is configured to *not* fsync() on all Append() calls
I20260619 15:05:51.715148 5920 tablet_bootstrap.cc:492] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071: No bootstrap required, opened a new log
I20260619 15:05:51.715649 5920 ts_tablet_manager.cc:1397] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.004s
I20260619 15:05:51.721419 5921 raft_consensus.cc:348] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c85b32c330a463b846c873af03eae28" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45423 } } peers { permanent_uuid: "43e979620e4e4ee58666a27cb54b7071" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35635 } } peers { permanent_uuid: "23daedc0b2224346a687872aa008eac0" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44771 } }
I20260619 15:05:51.721897 5921 raft_consensus.cc:374] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:05:51.721988 5921 raft_consensus.cc:729] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c85b32c330a463b846c873af03eae28, State: Initialized, Role: FOLLOWER
I20260619 15:05:51.723315 5921 consensus_queue.cc:260] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c85b32c330a463b846c873af03eae28" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45423 } } peers { permanent_uuid: "43e979620e4e4ee58666a27cb54b7071" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35635 } } peers { permanent_uuid: "23daedc0b2224346a687872aa008eac0" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44771 } }
I20260619 15:05:51.726016 5922 tablet_bootstrap.cc:492] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0: No bootstrap required, opened a new log
I20260619 15:05:51.726322 5922 ts_tablet_manager.cc:1397] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20260619 15:05:51.728997 5920 raft_consensus.cc:348] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c85b32c330a463b846c873af03eae28" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45423 } } peers { permanent_uuid: "43e979620e4e4ee58666a27cb54b7071" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35635 } } peers { permanent_uuid: "23daedc0b2224346a687872aa008eac0" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44771 } }
I20260619 15:05:51.729934 5920 raft_consensus.cc:374] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:05:51.730047 5920 raft_consensus.cc:729] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e979620e4e4ee58666a27cb54b7071, State: Initialized, Role: FOLLOWER
I20260619 15:05:51.730749 5920 consensus_queue.cc:260] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c85b32c330a463b846c873af03eae28" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45423 } } peers { permanent_uuid: "43e979620e4e4ee58666a27cb54b7071" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35635 } } peers { permanent_uuid: "23daedc0b2224346a687872aa008eac0" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44771 } }
I20260619 15:05:51.731920 5921 ts_tablet_manager.cc:1428] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28: Time spent starting tablet: real 0.021s user 0.008s sys 0.004s
I20260619 15:05:51.732045 5613 heartbeater.cc:503] Master 127.24.103.126:36547 was elected leader, sending a full tablet report...
I20260619 15:05:51.732671 5920 ts_tablet_manager.cc:1428] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260619 15:05:51.739596 5922 raft_consensus.cc:348] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c85b32c330a463b846c873af03eae28" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45423 } } peers { permanent_uuid: "43e979620e4e4ee58666a27cb54b7071" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35635 } } peers { permanent_uuid: "23daedc0b2224346a687872aa008eac0" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44771 } }
I20260619 15:05:51.740042 5922 raft_consensus.cc:374] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:05:51.740180 5922 raft_consensus.cc:729] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23daedc0b2224346a687872aa008eac0, State: Initialized, Role: FOLLOWER
I20260619 15:05:51.740797 5922 consensus_queue.cc:260] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c85b32c330a463b846c873af03eae28" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45423 } } peers { permanent_uuid: "43e979620e4e4ee58666a27cb54b7071" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35635 } } peers { permanent_uuid: "23daedc0b2224346a687872aa008eac0" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44771 } }
I20260619 15:05:51.741981 5892 heartbeater.cc:503] Master 127.24.103.126:36547 was elected leader, sending a full tablet report...
I20260619 15:05:51.742653 5922 ts_tablet_manager.cc:1428] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0: Time spent starting tablet: real 0.016s user 0.005s sys 0.009s
W20260619 15:05:51.757686 5897 tablet.cc:2369] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:05:51.785305 5928 raft_consensus.cc:482] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:05:51.785631 5928 raft_consensus.cc:504] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c85b32c330a463b846c873af03eae28" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45423 } } peers { permanent_uuid: "43e979620e4e4ee58666a27cb54b7071" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35635 } } peers { permanent_uuid: "23daedc0b2224346a687872aa008eac0" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44771 } }
I20260619 15:05:51.787401 5928 leader_election.cc:288] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c85b32c330a463b846c873af03eae28 (127.24.103.66:45423), 43e979620e4e4ee58666a27cb54b7071 (127.24.103.65:35635)
I20260619 15:05:51.794732 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.787876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:50849 (local address 127.24.103.65:35635)
0619 15:05:51.788141 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:51.788151 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:51.788678 (+ 527us) server_negotiation.cc:407] Connection header received
0619 15:05:51.788873 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:51.788877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:51.788933 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:51.789034 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:05:51.789736 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.790701 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.792700 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.793776 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.793934 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.794013 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:51.794342 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 15:05:51.794476 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 15:05:51.795284 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.787669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:45423 (local address 127.24.103.67:51611)
0619 15:05:51.788465 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:51.788484 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:51.788630 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:51.789006 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:51.789013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:51.789046 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:05:51.789260 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:51.789272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.790608 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.790614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:51.792920 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:51.792934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.794335 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.794341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:51.794517 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.794605 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 15:05:51.794697 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 15:05:51.794850 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":87}
I20260619 15:05:51.795583 5929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.787809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35635 (local address 127.24.103.67:50849)
0619 15:05:51.788563 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:51.788581 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:51.788723 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:51.789145 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:51.789149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:51.789172 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:05:51.789570 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:51.789580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.790862 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.790868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:51.792563 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:51.792574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.793974 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.793980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:51.794152 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.794186 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 15:05:51.794955 (+ 769us) client_negotiation.cc:240] Negotiation successful
0619 15:05:51.795099 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":3,"thread_start_us":441,"threads_started":1}
I20260619 15:05:51.796053 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.788142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:51611 (local address 127.24.103.66:45423)
0619 15:05:51.788330 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:51.788336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:51.788743 (+ 407us) server_negotiation.cc:407] Connection header received
0619 15:05:51.788819 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:51.788824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:51.788893 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:51.788974 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:05:51.789396 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.790429 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.793084 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.794192 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.794342 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.794406 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:51.794733 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 15:05:51.795311 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 15:05:51.796653 5568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3474b2967843839bc30d54be5a80b6" candidate_uuid: "23daedc0b2224346a687872aa008eac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e979620e4e4ee58666a27cb54b7071" is_pre_election: true
I20260619 15:05:51.797101 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3474b2967843839bc30d54be5a80b6" candidate_uuid: "23daedc0b2224346a687872aa008eac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c85b32c330a463b846c873af03eae28" is_pre_election: true
I20260619 15:05:51.797351 5568 raft_consensus.cc:2454] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23daedc0b2224346a687872aa008eac0 in term 0.
I20260619 15:05:51.797676 5712 raft_consensus.cc:2454] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23daedc0b2224346a687872aa008eac0 in term 0.
I20260619 15:05:51.798113 5781 leader_election.cc:302] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23daedc0b2224346a687872aa008eac0, 43e979620e4e4ee58666a27cb54b7071; no voters:
I20260619 15:05:51.798559 5928 raft_consensus.cc:2781] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:05:51.798698 5928 raft_consensus.cc:482] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:05:51.798810 5928 raft_consensus.cc:3037] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:05:51.801823 5928 raft_consensus.cc:504] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c85b32c330a463b846c873af03eae28" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45423 } } peers { permanent_uuid: "43e979620e4e4ee58666a27cb54b7071" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35635 } } peers { permanent_uuid: "23daedc0b2224346a687872aa008eac0" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44771 } }
I20260619 15:05:51.802516 5928 leader_election.cc:288] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [CANDIDATE]: Term 1 election: Requested vote from peers 4c85b32c330a463b846c873af03eae28 (127.24.103.66:45423), 43e979620e4e4ee58666a27cb54b7071 (127.24.103.65:35635)
I20260619 15:05:51.803445 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3474b2967843839bc30d54be5a80b6" candidate_uuid: "23daedc0b2224346a687872aa008eac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c85b32c330a463b846c873af03eae28"
I20260619 15:05:51.803681 5712 raft_consensus.cc:3037] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:05:51.806862 5712 raft_consensus.cc:2454] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23daedc0b2224346a687872aa008eac0 in term 1.
I20260619 15:05:51.807399 5568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3474b2967843839bc30d54be5a80b6" candidate_uuid: "23daedc0b2224346a687872aa008eac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e979620e4e4ee58666a27cb54b7071"
I20260619 15:05:51.807613 5568 raft_consensus.cc:3037] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:05:51.807642 5780 leader_election.cc:302] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23daedc0b2224346a687872aa008eac0, 4c85b32c330a463b846c873af03eae28; no voters:
I20260619 15:05:51.807945 5928 raft_consensus.cc:2781] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:05:51.810973 5568 raft_consensus.cc:2454] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23daedc0b2224346a687872aa008eac0 in term 1.
I20260619 15:05:51.811904 5928 raft_consensus.cc:686] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 1 LEADER]: Becoming Leader. State: Replica: 23daedc0b2224346a687872aa008eac0, State: Running, Role: LEADER
I20260619 15:05:51.812479 5928 consensus_queue.cc:237] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c85b32c330a463b846c873af03eae28" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45423 } } peers { permanent_uuid: "43e979620e4e4ee58666a27cb54b7071" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35635 } } peers { permanent_uuid: "23daedc0b2224346a687872aa008eac0" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44771 } }
I20260619 15:05:51.817711 5415 catalog_manager.cc:5495] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23daedc0b2224346a687872aa008eac0 (127.24.103.67). New cstate: current_term: 1 leader_uuid: "23daedc0b2224346a687872aa008eac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c85b32c330a463b846c873af03eae28" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e979620e4e4ee58666a27cb54b7071" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23daedc0b2224346a687872aa008eac0" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44771 } health_report { overall_health: HEALTHY } } }
W20260619 15:05:51.825110 5758 tablet.cc:2369] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:05:51.829620 5614 tablet.cc:2369] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:05:51.862672 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.853185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.24.103.67:44771)
0619 15:05:51.853671 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:51.853677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:51.853748 (+ 71us) server_negotiation.cc:407] Connection header received
0619 15:05:51.855198 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:51.855207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:51.855352 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:51.855524 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:05:51.856056 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.857184 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.859539 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.860171 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.860309 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.861050 (+ 741us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:05:51.861972 (+ 922us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:05:51.862225 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:51.862278 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 15:05:51.862372 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
I20260619 15:05:51.872131 5568 raft_consensus.cc:1261] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071 [term 1 FOLLOWER]: Refusing update from remote peer 23daedc0b2224346a687872aa008eac0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:05:51.872278 5712 raft_consensus.cc:1261] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28 [term 1 FOLLOWER]: Refusing update from remote peer 23daedc0b2224346a687872aa008eac0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:05:51.873261 5928 consensus_queue.cc:1035] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e979620e4e4ee58666a27cb54b7071" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:05:51.873557 5930 consensus_queue.cc:1035] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c85b32c330a463b846c873af03eae28" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:05:51.912292 5416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:05:51.912634 5416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:05:51.917186 5416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 787e4047bab34150a2c6380a22f461c9: Sending DeleteTablet for 3 replicas of tablet 6f3474b2967843839bc30d54be5a80b6
I20260619 15:05:51.918547 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f3474b2967843839bc30d54be5a80b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:05:51 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:48450
I20260619 15:05:51.919338 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f3474b2967843839bc30d54be5a80b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:05:51 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:33782
I20260619 15:05:51.921087 5827 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f3474b2967843839bc30d54be5a80b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:05:51 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:34506
I20260619 15:05:51.931674 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 5453
I20260619 15:05:51.935384 5942 tablet_replica.cc:330] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071: stopping tablet replica
I20260619 15:05:51.936023 5942 raft_consensus.cc:2229] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:05:51.936699 5942 raft_consensus.cc:2258] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:05:51.937692 5944 tablet_replica.cc:330] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0: stopping tablet replica
I20260619 15:05:51.938560 5944 raft_consensus.cc:2229] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:05:51.939769 5944 raft_consensus.cc:2258] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:05:51.947503 5942 ts_tablet_manager.cc:1905] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:05:51.956867 5943 tablet_replica.cc:330] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28: stopping tablet replica
I20260619 15:05:51.958958 5942 ts_tablet_manager.cc:1918] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:05:51.959313 5942 log.cc:1196] T 6f3474b2967843839bc30d54be5a80b6 P 43e979620e4e4ee58666a27cb54b7071: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/6f3474b2967843839bc30d54be5a80b6
I20260619 15:05:51.959393 5943 raft_consensus.cc:2229] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:05:51.959721 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:51.932680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.24.103.65:35635)
0619 15:05:51.933233 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:51.933239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:51.933258 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:05:51.933319 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:51.933324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:51.933397 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:51.933486 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:51.939177 (+ 5691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.940317 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.945849 (+ 5532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:51.946553 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:51.946683 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:51.950471 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:51.950488 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:51.950508 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:51.950584 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:51.954944 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:51.955941 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:51.955951 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:51.955955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:51.956116 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:51.956623 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:51.956628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:51.956632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:51.957185 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:51.958602 (+ 1417us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:51.958850 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 15:05:51.959103 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418}
I20260619 15:05:51.959985 5943 raft_consensus.cc:2258] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:05:51.960435 5608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:05:51.962551 5943 ts_tablet_manager.cc:1905] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:05:51.968259 5944 ts_tablet_manager.cc:1905] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:05:51.969631 5943 ts_tablet_manager.cc:1918] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:05:51.969816 5943 log.cc:1196] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/6f3474b2967843839bc30d54be5a80b6
I20260619 15:05:51.970343 5943 ts_tablet_manager.cc:1939] T 6f3474b2967843839bc30d54be5a80b6 P 4c85b32c330a463b846c873af03eae28: Deleting consensus metadata
I20260619 15:05:51.972363 5401 catalog_manager.cc:4841] TS 4c85b32c330a463b846c873af03eae28 (127.24.103.66:45423): tablet 6f3474b2967843839bc30d54be5a80b6 (table test-table [id=54182d0535db43ed9d34d843026a376c]) successfully deleted
I20260619 15:05:51.978899 5944 ts_tablet_manager.cc:1918] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:05:51.979107 5944 log.cc:1196] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/6f3474b2967843839bc30d54be5a80b6
I20260619 15:05:51.979676 5944 ts_tablet_manager.cc:1939] T 6f3474b2967843839bc30d54be5a80b6 P 23daedc0b2224346a687872aa008eac0: Deleting consensus metadata
I20260619 15:05:51.981608 5401 catalog_manager.cc:4841] TS 23daedc0b2224346a687872aa008eac0 (127.24.103.67:44771): tablet 6f3474b2967843839bc30d54be5a80b6 (table test-table [id=54182d0535db43ed9d34d843026a376c]) successfully deleted
I20260619 15:05:52.128080 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 5453
W20260619 15:05:52.151684 5402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.103.65:35635 (error 108)
I20260619 15:05:52.151849 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 5615
I20260619 15:05:52.154162 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:52.153627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35635 (local address 127.0.0.1:48474)
0619 15:05:52.153852 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:52.153973 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:35635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260619 15:05:52.154393 5402 catalog_manager.cc:4584] TS 43e979620e4e4ee58666a27cb54b7071 (127.24.103.65:35635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f3474b2967843839bc30d54be5a80b6: Network error: Client connection negotiation failed: client connection to 127.24.103.65:35635: connect: Connection refused (error 111)
I20260619 15:05:52.171361 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:52.153172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.24.103.66:45423)
0619 15:05:52.153351 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:52.153356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:52.153376 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:05:52.153438 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:52.153442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:52.153514 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:52.153628 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:52.155625 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:52.156774 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:52.158720 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:52.159345 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:52.159467 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:52.163833 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:52.163851 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:52.163870 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:52.163925 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:52.167995 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:52.168970 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:52.168983 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:52.168988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:52.169156 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:52.169579 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:52.169584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:52.169587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:52.170046 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:52.170364 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:52.170687 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 15:05:52.170975 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:05:52.172118 5752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:05:52.208045 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:52.207482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35635 (local address 127.0.0.1:48488)
0619 15:05:52.207776 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:52.207880 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:35635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260619 15:05:52.209141 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:52.208743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35635 (local address 127.0.0.1:48498)
0619 15:05:52.208911 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:52.208978 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:35635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 15:05:52.269830 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:52.269295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35635 (local address 127.0.0.1:48504)
0619 15:05:52.269555 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:52.269651 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:35635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260619 15:05:52.271010 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:52.270555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35635 (local address 127.0.0.1:48508)
0619 15:05:52.270745 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:52.270861 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:35635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 15:05:52.328294 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 5615
I20260619 15:05:52.351718 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 5762
I20260619 15:05:52.352275 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:52.351136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35635 (local address 127.0.0.1:48524)
0619 15:05:52.351991 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:52.352099 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:35635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":716}
I20260619 15:05:52.354794 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:52.354265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35635 (local address 127.0.0.1:48530)
0619 15:05:52.354509 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:52.354603 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:35635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260619 15:05:52.373167 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:52.353078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.24.103.67:44771)
0619 15:05:52.353253 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:52.353259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:52.353278 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:05:52.353341 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:52.353346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:52.353426 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:52.353533 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:52.355480 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:52.356744 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:52.358686 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:52.359640 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:52.359799 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:52.363594 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:52.363614 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:52.363638 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:52.363707 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:52.369605 (+ 5898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:52.370616 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:52.370626 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:52.370630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:52.370867 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:52.371354 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:52.371359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:52.371363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:52.371846 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:52.372128 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:52.372485 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 15:05:52.372727 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 15:05:52.373939 5887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:05:52.505741 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 5762
I20260619 15:05:52.525411 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 5383
I20260619 15:05:52.526649 5446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:05:52.636597 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:52.526837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35635 (local address 127.0.0.1:48544)
0619 15:05:52.636317 (+109480us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:52.636426 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:35635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 15:05:52.636924 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 5383
I20260619 15:05:52.658509 24989 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 15:05:52.994694 24989 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 15:05:52.995014 24989 mini_ranger.cc:66] Stopping Ranger...
I20260619 15:05:53.341570 24989 mini_ranger.cc:68] Stopped Ranger
2026-06-19 15:05:53.341 UTC [4435] LOG: received smart shutdown request
2026-06-19 15:05:53.344 UTC [4435] LOG: background worker "logical replication launcher" (PID 4444) exited with exit code 1
2026-06-19 15:05:53.345 UTC [4439] LOG: shutting down
2026-06-19 15:05:53.501 UTC [4435] LOG: database system is shut down
2026-06-19T15:05:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53750 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:05:53 dist-test-slave-zpbw krb5kdc[5958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:05:53 dist-test-slave-zpbw krb5kdc[5958](info): set up 2 sockets
Jun 19 15:05:53 dist-test-slave-zpbw krb5kdc[5958](info): commencing operation
krb5kdc: starting...
W20260619 15:05:54.727255 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:05:54 dist-test-slave-zpbw krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:05:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:05:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:05:54.959378 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:46673
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:46673,127.24.103.125:40973,127.24.103.124:44139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:05:55.290407 5971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:05:55.290716 5971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:05:55.290892 5971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:05:55.298202 5971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:05:55.298324 5971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:05:55.298373 5971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:05:55.298450 5971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:05:55.298518 5971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:05:55.308713 5971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:46673,127.24.103.125:40973,127.24.103.124:44139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:46673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:05:55.310179 5971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:05:55.311578 5971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:05:55.323796 5979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:55.323768 5976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:55.323750 5977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:55.325012 5971 server_base.cc:1034] running on GCE node
I20260619 15:05:55.325848 5971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:05:55.327384 5971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:05:55.328621 5971 hybrid_clock.cc:648] HybridClock initialized: now 1781881555328582 us; error 49 us; skew 500 ppm
I20260619 15:05:55.328985 5971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:05:55 dist-test-slave-zpbw krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:55.332123 5971 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:05:55.333664 5971 webserver.cc:460] Webserver started at http://127.24.103.126:39769/ using document root <none> and password file <none>
I20260619 15:05:55.334367 5971 fs_manager.cc:359] Metadata directory not provided
I20260619 15:05:55.334512 5971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:05:55.334874 5971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:05:55.337913 5971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "f37b193d85da4b90b249aefa7562e6b5"
format_stamp: "Formatted at 2026-06-19 15:05:55 on dist-test-slave-zpbw"
server_key: "7133cf04afb6b1d8df9056cb4b69fc49"
server_key_iv: "ddef09e4209ad2c0a71295b2dd86d635"
server_key_version: "encryptionkey@0"
I20260619 15:05:55.338979 5971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "f37b193d85da4b90b249aefa7562e6b5"
format_stamp: "Formatted at 2026-06-19 15:05:55 on dist-test-slave-zpbw"
server_key: "7133cf04afb6b1d8df9056cb4b69fc49"
server_key_iv: "ddef09e4209ad2c0a71295b2dd86d635"
server_key_version: "encryptionkey@0"
I20260619 15:05:55.344274 5971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 15:05:55.347759 5986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:55.349128 5971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:05:55.349313 5971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "f37b193d85da4b90b249aefa7562e6b5"
format_stamp: "Formatted at 2026-06-19 15:05:55 on dist-test-slave-zpbw"
server_key: "7133cf04afb6b1d8df9056cb4b69fc49"
server_key_iv: "ddef09e4209ad2c0a71295b2dd86d635"
server_key_version: "encryptionkey@0"
I20260619 15:05:55.349534 5971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:05:55.363921 5971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:05:55.368116 5971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:05:55.368453 5971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:05:55.388890 5971 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:46673
I20260619 15:05:55.388912 6037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:46673 every 8 connection(s)
I20260619 15:05:55.390404 5971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:05:55.392499 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 5971
I20260619 15:05:55.392725 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:05:55.393414 24989 external_mini_cluster.cc:1428] Setting key 5b19e52e859c9bf2f5ba7ce16143d663
I20260619 15:05:55.394555 6038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:05:55.401551 6038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.103.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.125@KRBTEST.COM" created.
Jun 19 15:05:55 dist-test-slave-zpbw krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:05:55.429065 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.405043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:46673 (local address 127.0.0.1:57244)
0619 15:05:55.406216 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.406280 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:55.409850 (+ 3570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:55.410980 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:55.410985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:55.411585 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:55.411939 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:55.411956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:55.414409 (+ 2453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:55.414413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:55.415720 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:55.415739 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:55.416972 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:55.416979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:55.417149 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:55.417888 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:55.417911 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:55.420470 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:55.424540 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:55.424554 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:55.424572 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:55.425006 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:55.425434 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:55.425440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:55.425444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:55.425609 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:55.426433 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:55.426446 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:55.426683 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 15:05:55.427000 (+ 317us) client_negotiation.cc:240] Negotiation successful
0619 15:05:55.427634 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":182,"thread_start_us":532,"threads_started":1}
I20260619 15:05:55.429083 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.405043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.24.103.126:46673)
0619 15:05:55.409991 (+ 4948us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:55.410003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:55.410064 (+ 61us) server_negotiation.cc:407] Connection header received
0619 15:05:55.410280 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:55.410292 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:55.410682 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:55.410986 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:55.412150 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:55.414241 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:55.415869 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:55.416715 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:55.416934 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:55.420720 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:55.420730 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:55.420744 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:55.420774 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:55.424372 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:55.425146 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:55.425155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:55.425159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:55.425323 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:55.425721 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:55.425726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:55.425729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:55.426288 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:55.426516 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:55.427132 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 15:05:55.427528 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":4667,"spinlock_wait_cycles":39808,"thread_start_us":4329,"threads_started":1}
I20260619 15:05:55.432471 6038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:05:55.433894 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.433346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:34480)
0619 15:05:55.433564 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.433709 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 15:05:55.434235 5989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
I20260619 15:05:55.435989 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.435513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:34488)
0619 15:05:55.435718 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.435813 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 15:05:55.436192 6038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:40973: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.125.keytab.
Entry for principal kudu/127.24.103.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.125.keytab.
I20260619 15:05:55.444199 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:40973
--webserver_interface=127.24.103.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.125.keytab
--principal=kudu/127.24.103.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:46673,127.24.103.125:40973,127.24.103.124:44139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:05:55.485594 6038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } attempt: 1
I20260619 15:05:55.486935 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.486327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:34496)
0619 15:05:55.486584 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.486733 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 15:05:55.488101 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.487669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:34508)
0619 15:05:55.487842 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.487934 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 15:05:55.488314 6038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:40973: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
I20260619 15:05:55.556597 6038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } attempt: 2
I20260619 15:05:55.558040 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.557339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:34518)
0619 15:05:55.557636 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.557778 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 15:05:55.559157 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.558676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:34534)
0619 15:05:55.558880 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.558986 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 15:05:55.559422 6038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:40973: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
I20260619 15:05:55.650769 6038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } attempt: 3
I20260619 15:05:55.652251 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.651616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:34538)
0619 15:05:55.651871 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.652035 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 15:05:55.653379 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.652982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:34552)
0619 15:05:55.653134 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.653211 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 15:05:55.653625 6038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:40973: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
W20260619 15:05:55.788771 6045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:05:55.789067 6045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:05:55.789157 6045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:05:55.796691 6045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:05:55.796845 6045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:05:55.796947 6045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:05:55.797034 6045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260619 15:05:55.796923 6038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } attempt: 4
W20260619 15:05:55.797099 6045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:05:55.798128 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.797560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:34562)
0619 15:05:55.797888 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.797987 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260619 15:05:55.799182 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.798836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:34572)
0619 15:05:55.798962 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.799058 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 15:05:55.799427 6038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:40973: Network error: Client connection negotiation failed: client connection to 127.24.103.125:40973: connect: Connection refused (error 111)
I20260619 15:05:55.807536 6045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:46673,127.24.103.125:40973,127.24.103.124:44139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.125.keytab
--principal=kudu/127.24.103.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.103.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:05:55.808861 6045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:05:55.810155 6045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:05:55.822101 6050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:55.822187 6051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:55.822391 6053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:55.823333 6045 server_base.cc:1034] running on GCE node
I20260619 15:05:55.824150 6045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:05:55.825627 6045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:05:55.826911 6045 hybrid_clock.cc:648] HybridClock initialized: now 1781881555826854 us; error 42 us; skew 500 ppm
I20260619 15:05:55.827278 6045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:05:55 dist-test-slave-zpbw krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:55.830438 6045 init.cc:377] Logged in from keytab as kudu/127.24.103.125@KRBTEST.COM (short username kudu)
I20260619 15:05:55.832046 6045 webserver.cc:460] Webserver started at http://127.24.103.125:42637/ using document root <none> and password file <none>
I20260619 15:05:55.832760 6045 fs_manager.cc:359] Metadata directory not provided
I20260619 15:05:55.832911 6045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:05:55.833217 6045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:05:55.836385 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/data/instance:
uuid: "88bd07e8bfbc48aa8c2845cd66d9220f"
format_stamp: "Formatted at 2026-06-19 15:05:55 on dist-test-slave-zpbw"
server_key: "3aaff1d00a5ae4092fc1860dcbdf6ac6"
server_key_iv: "621749a4f37fbebb7049ca8ee1c51a0f"
server_key_version: "encryptionkey@0"
I20260619 15:05:55.837234 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/wal/instance:
uuid: "88bd07e8bfbc48aa8c2845cd66d9220f"
format_stamp: "Formatted at 2026-06-19 15:05:55 on dist-test-slave-zpbw"
server_key: "3aaff1d00a5ae4092fc1860dcbdf6ac6"
server_key_iv: "621749a4f37fbebb7049ca8ee1c51a0f"
server_key_version: "encryptionkey@0"
I20260619 15:05:55.842684 6045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 15:05:55.846401 6060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:55.848016 6045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 15:05:55.848222 6045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/wal
uuid: "88bd07e8bfbc48aa8c2845cd66d9220f"
format_stamp: "Formatted at 2026-06-19 15:05:55 on dist-test-slave-zpbw"
server_key: "3aaff1d00a5ae4092fc1860dcbdf6ac6"
server_key_iv: "621749a4f37fbebb7049ca8ee1c51a0f"
server_key_version: "encryptionkey@0"
I20260619 15:05:55.848433 6045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:05:55.862663 6045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:05:55.866993 6045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:05:55.867311 6045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:05:55.886013 6045 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.125:40973
I20260619 15:05:55.886085 6111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.125:40973 every 8 connection(s)
I20260619 15:05:55.887477 6045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
I20260619 15:05:55.891332 6112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:05:55.896653 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 6045
I20260619 15:05:55.896849 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-1/wal/instance
I20260619 15:05:55.897354 24989 external_mini_cluster.cc:1428] Setting key 1085dbfa2070ce2305ebac27e1f540ec
I20260619 15:05:55.898537 6112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:05:55 dist-test-slave-zpbw krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.124@KRBTEST.COM" created.
I20260619 15:05:55.921023 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.902053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.24.103.126:46673)
0619 15:05:55.902272 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:55.902279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:55.903664 (+ 1385us) server_negotiation.cc:407] Connection header received
0619 15:05:55.906226 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:55.906232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:55.906326 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:55.906453 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:55.908301 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:55.908968 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:55.910529 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:55.911073 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:55.911210 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:55.914766 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:55.914804 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:55.914812 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:55.914856 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:55.917486 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:55.918374 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:55.918378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:55.918381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:55.918503 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:55.919014 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:55.919018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:55.919020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:55.919321 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:55.919482 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:55.920346 (+ 864us) server_negotiation.cc:299] Negotiation successful
0619 15:05:55.920618 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 15:05:55.921913 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.902559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:46673 (local address 127.0.0.1:57252)
0619 15:05:55.903451 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.903517 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:55.905931 (+ 2414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:55.906696 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:55.906705 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:55.907304 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:55.908130 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:55.908150 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:55.909139 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:55.909144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:55.910380 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:55.910392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:55.911304 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:55.911311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:55.911550 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:55.912264 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:55.912287 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:55.914519 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:55.917652 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:55.917674 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:55.917696 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:55.918220 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:55.918631 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:55.918635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:55.918639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:55.918914 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:55.919479 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:55.919489 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:55.919902 (+ 413us) client_negotiation.cc:769] Sending connection context
0619 15:05:55.920239 (+ 337us) client_negotiation.cc:240] Negotiation successful
0619 15:05:55.920693 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":387,"threads_started":1}
I20260619 15:05:55.923734 6112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:05:55 dist-test-slave-zpbw krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260619 15:05:55.940676 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.924580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:34582)
0619 15:05:55.924791 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.924818 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:55.924948 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:55.925797 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:55.925800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:55.926179 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:55.926317 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:55.926325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:55.928055 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:55.928060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:55.929580 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:55.929598 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:55.930401 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:55.930409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:55.930671 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:55.931337 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:55.931364 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:55.934226 (+ 2862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:55.937710 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:55.937718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:55.937725 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:55.938294 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:55.938670 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:55.938678 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:55.938683 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:55.938918 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:55.939535 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:55.939543 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:55.939715 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 15:05:55.939875 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 15:05:55.940221 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 15:05:55.940675 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.924581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.24.103.125:40973)
0619 15:05:55.925235 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:55.925245 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:55.925277 (+ 32us) server_negotiation.cc:407] Connection header received
0619 15:05:55.925335 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:55.925345 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:55.925632 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:55.925776 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:55.926464 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:55.927915 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:55.929720 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:55.930251 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:55.930343 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:55.934494 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:55.934507 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:55.934529 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:55.934563 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:55.937497 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:55.938421 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:55.938428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:55.938431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:55.938556 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:55.939043 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:55.939046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:55.939048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:55.939403 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:55.939572 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:55.940018 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 15:05:55.940325 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":452,"thread_start_us":269,"threads_started":1}
I20260619 15:05:55.943187 6112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:05:55.944573 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.944038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46254)
0619 15:05:55.944209 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.944354 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 15:05:55.944887 6062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
I20260619 15:05:55.946436 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.946090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46266)
0619 15:05:55.946231 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.946306 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 15:05:55.946691 6112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44139: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.124.keytab.
Entry for principal kudu/127.24.103.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.124.keytab.
I20260619 15:05:55.949988 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:44139
--webserver_interface=127.24.103.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.124.keytab
--principal=kudu/127.24.103.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:46673,127.24.103.125:40973,127.24.103.124:44139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:05:55.996111 6112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } attempt: 1
I20260619 15:05:55.997515 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.996854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46274)
0619 15:05:55.997127 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.997286 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 15:05:55.998708 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:55.998234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46290)
0619 15:05:55.998431 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:55.998539 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 15:05:55.998960 6112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44139: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
I20260619 15:05:56.067236 6112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } attempt: 2
I20260619 15:05:56.068471 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.067922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46306)
0619 15:05:56.068160 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.068278 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 15:05:56.069480 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.069119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46316)
0619 15:05:56.069264 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.069346 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 15:05:56.069701 6112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44139: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
I20260619 15:05:56.098693 6038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } attempt: 5
Jun 19 15:05:56 dist-test-slave-zpbw krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260619 15:05:56.113492 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.099437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:34596)
0619 15:05:56.099697 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.099718 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:56.099849 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:56.100222 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:56.100227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:56.100725 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:56.100964 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:56.100977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.101951 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.101957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:56.103524 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:56.103540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.104287 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.104293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:56.104476 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.105206 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:56.105237 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:56.107929 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:56.111015 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:56.111028 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:56.111033 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:56.111443 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:56.111838 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:56.111844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:56.111847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:56.111989 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:56.112514 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:56.112519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:56.112679 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 15:05:56.112824 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 15:05:56.113134 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 15:05:56.113480 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.099606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.24.103.125:40973)
0619 15:05:56.099761 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:56.099765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:56.099783 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:05:56.099980 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:56.099984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:56.100065 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:56.100150 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:56.101136 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.101809 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.103685 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.104138 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.104234 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.108142 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:56.108148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:56.108151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:56.108183 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:56.110866 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.111586 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.111592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.111596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.111726 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.112101 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.112108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.112111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.112401 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:56.112560 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:56.112926 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 15:05:56.113147 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 15:05:56.114507 6038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:05:56.115742 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.115383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46326)
0619 15:05:56.115545 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.115619 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 15:05:56.116648 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.116337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46336)
0619 15:05:56.116465 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.116550 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 15:05:56.116873 6038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44139: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
I20260619 15:05:56.160961 6112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } attempt: 3
I20260619 15:05:56.162298 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.161798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46342)
0619 15:05:56.162033 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.162138 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 15:05:56.163575 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.163150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46356)
0619 15:05:56.163377 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.163455 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260619 15:05:56.163877 6112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44139: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
I20260619 15:05:56.168077 6038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } attempt: 1
I20260619 15:05:56.169209 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.168619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46362)
0619 15:05:56.168839 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.168969 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 15:05:56.170323 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.169938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46372)
0619 15:05:56.170108 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.170198 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 15:05:56.170570 6038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44139: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
I20260619 15:05:56.238916 6038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } attempt: 2
I20260619 15:05:56.240193 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.239576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46386)
0619 15:05:56.239844 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.239966 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 15:05:56.241295 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.240870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46400)
0619 15:05:56.241051 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.241146 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 15:05:56.241561 6038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44139: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
W20260619 15:05:56.289445 6119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:05:56.289831 6119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:05:56.289995 6119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:05:56.297555 6119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:05:56.297720 6119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:05:56.297835 6119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:05:56.297930 6119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:05:56.298012 6119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:05:56.307240 6112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } attempt: 4
I20260619 15:05:56.308475 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.307984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46414)
0619 15:05:56.308217 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.308315 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 15:05:56.308761 6119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:46673,127.24.103.125:40973,127.24.103.124:44139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.124.keytab
--principal=kudu/127.24.103.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:44139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.103.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:05:56.309584 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.309271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46430)
0619 15:05:56.309400 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.309468 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 15:05:56.309928 6112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44139: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
I20260619 15:05:56.310432 6119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:05:56.311832 6119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:05:56.323827 6124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:56.323736 6125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:56.323771 6127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:56.324954 6119 server_base.cc:1034] running on GCE node
I20260619 15:05:56.325826 6119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:05:56.327419 6119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:05:56.328616 6119 hybrid_clock.cc:648] HybridClock initialized: now 1781881556328569 us; error 34 us; skew 500 ppm
I20260619 15:05:56.328991 6119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:05:56 dist-test-slave-zpbw krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881556, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:56.335650 6119 init.cc:377] Logged in from keytab as kudu/127.24.103.124@KRBTEST.COM (short username kudu)
I20260619 15:05:56.337426 6119 webserver.cc:460] Webserver started at http://127.24.103.124:37535/ using document root <none> and password file <none>
I20260619 15:05:56.338177 6119 fs_manager.cc:359] Metadata directory not provided
I20260619 15:05:56.338320 6119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:05:56.338678 6119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:05:56.341825 6119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/data/instance:
uuid: "340bf1664d0b485692237518e295bcd6"
format_stamp: "Formatted at 2026-06-19 15:05:56 on dist-test-slave-zpbw"
server_key: "5329faf98f1f620df3ac973d415ca76f"
server_key_iv: "fc38a19f5cb13e518af58869a2907500"
server_key_version: "encryptionkey@0"
I20260619 15:05:56.342732 6119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/wal/instance:
uuid: "340bf1664d0b485692237518e295bcd6"
format_stamp: "Formatted at 2026-06-19 15:05:56 on dist-test-slave-zpbw"
server_key: "5329faf98f1f620df3ac973d415ca76f"
server_key_iv: "fc38a19f5cb13e518af58869a2907500"
server_key_version: "encryptionkey@0"
I20260619 15:05:56.347841 6038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } attempt: 3
I20260619 15:05:56.348630 6119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 15:05:56.349067 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.348509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46432)
0619 15:05:56.348769 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.348870 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 15:05:56.350175 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.349732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:46434)
0619 15:05:56.349921 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.350036 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260619 15:05:56.350445 6038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44139: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44139: connect: Connection refused (error 111)
I20260619 15:05:56.352927 6134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:56.354411 6119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 15:05:56.354585 6119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/wal
uuid: "340bf1664d0b485692237518e295bcd6"
format_stamp: "Formatted at 2026-06-19 15:05:56 on dist-test-slave-zpbw"
server_key: "5329faf98f1f620df3ac973d415ca76f"
server_key_iv: "fc38a19f5cb13e518af58869a2907500"
server_key_version: "encryptionkey@0"
I20260619 15:05:56.354784 6119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:05:56.385643 6119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:05:56.390061 6119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:05:56.390367 6119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:05:56.409224 6119 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.124:44139
I20260619 15:05:56.409246 6185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.124:44139 every 8 connection(s)
I20260619 15:05:56.410723 6119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
I20260619 15:05:56.414438 6186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:05:56.415071 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 6119
I20260619 15:05:56.415292 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881351219607-24989-0/minicluster-data/master-2/wal/instance
I20260619 15:05:56.415756 24989 external_mini_cluster.cc:1428] Setting key 7903d0d3a5354827d986bd176b768d45
I20260619 15:05:56.423693 6186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:05:56 dist-test-slave-zpbw krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
Jun 19 15:05:56 dist-test-slave-zpbw krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881556, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:05:56.440878 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.417706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.24.103.124:44139)
0619 15:05:56.418546 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:56.418557 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:56.418598 (+ 41us) server_negotiation.cc:407] Connection header received
0619 15:05:56.420471 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:56.420501 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:56.420932 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:56.421255 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:56.423156 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.424491 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.426434 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.427576 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.428456 (+ 880us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.432657 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:56.432670 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:56.432685 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:56.432744 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:56.436051 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.436815 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.436828 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.436835 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.436976 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.437426 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.437430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.437432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.438201 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:56.438544 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:56.439033 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 15:05:56.439485 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":388,"threads_started":1}
I20260619 15:05:56.442874 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.426521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:46673 (local address 127.0.0.1:57262)
0619 15:05:56.427431 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.427500 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:56.427694 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:56.428404 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:56.428413 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:56.429115 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:56.429365 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:56.429373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.430426 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.430431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:56.432157 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:56.432176 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.433145 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.433149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:56.433283 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.433997 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:56.434014 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:56.436486 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:56.440174 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:56.440184 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:56.440189 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:56.440597 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:56.441019 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:56.441039 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:56.441044 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:56.441183 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:56.441712 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:56.441719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:56.441893 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 15:05:56.442092 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 15:05:56.442425 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":355,"threads_started":1}
I20260619 15:05:56.442871 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.426577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.24.103.126:46673)
0619 15:05:56.427957 (+ 1380us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:56.427965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:56.428011 (+ 46us) server_negotiation.cc:407] Connection header received
0619 15:05:56.428072 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:56.428077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:56.428160 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:56.428278 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:56.429533 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.430273 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.432319 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.432950 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.433103 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.436678 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:56.436683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:56.436686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:56.436720 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:56.440029 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.440726 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.440731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.440735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.440906 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.441303 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.441308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.441310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.441581 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:56.441727 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:56.442201 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 15:05:56.442425 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":1130,"threads_started":1}
I20260619 15:05:56.444581 6186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:05:56 dist-test-slave-zpbw krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881556, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
Jun 19 15:05:56 dist-test-slave-zpbw krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:05:56.459478 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.445282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:40973 (local address 127.0.0.1:48522)
0619 15:05:56.445476 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.445496 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:56.445597 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:56.445888 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:56.445892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:56.446359 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:56.446496 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:56.446505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.447776 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.447781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:56.449153 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:56.449164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.450104 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.450111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:56.450342 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.450736 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:56.450758 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:56.453570 (+ 2812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:56.456813 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:56.456820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:56.456824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:56.457368 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:56.457710 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:56.457716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:56.457720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:56.457891 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:56.458449 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:56.458454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:56.458648 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 15:05:56.458824 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 15:05:56.459074 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 15:05:56.459558 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.445428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.24.103.125:40973)
0619 15:05:56.445582 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:56.445586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:56.445602 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:05:56.445707 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:56.445710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:56.445766 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:56.445840 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:56.446642 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.447653 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.449294 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.449947 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.450067 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.453818 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:56.453826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:56.453831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:56.453880 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:56.456685 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.457479 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.457483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.457486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.457621 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.457990 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.457993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.458000 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.458335 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:56.458491 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:56.458953 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 15:05:56.459192 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 15:05:56.460858 6186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:05:56.461156 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.444012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.24.103.126:46673)
0619 15:05:56.444169 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:56.444174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:56.444265 (+ 91us) server_negotiation.cc:407] Connection header received
0619 15:05:56.444439 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:56.444444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:56.444523 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:56.444621 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:56.446287 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.447323 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.449073 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.449663 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.449796 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.455047 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:56.455052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:56.455056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:56.455115 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:56.458271 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.459183 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.459190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.459193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.459332 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.459744 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.459748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.459752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.460064 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:56.460315 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:56.460663 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 15:05:56.460886 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 15:05:56 dist-test-slave-zpbw krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881556, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
Jun 19 15:05:56 dist-test-slave-zpbw krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260619 15:05:56.474581 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.461840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.24.103.124:44139)
0619 15:05:56.461992 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:56.462002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:56.462221 (+ 219us) server_negotiation.cc:407] Connection header received
0619 15:05:56.462387 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:56.462391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:56.462449 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:56.462540 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:56.463306 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.463971 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.465113 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.465522 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.465651 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.468806 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:56.468814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:56.468821 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:56.468860 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:56.471915 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.472688 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.472695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.472700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.472843 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.473271 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.473274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.473276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.473571 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:56.473680 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:56.474139 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 15:05:56.474313 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 15:05:56.474982 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.461998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:41122)
0619 15:05:56.462155 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.462175 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:56.462272 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:56.462554 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:56.462558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:56.463038 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:56.463183 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:56.463191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.464094 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.464101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:56.464984 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:56.464995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.465665 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.465670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:56.465838 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.466403 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:56.466420 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:56.468609 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:56.472049 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:56.472055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:56.472058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:56.472556 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:56.472963 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:56.472969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:56.472976 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:56.473150 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:56.473687 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:56.473695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:56.473859 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 15:05:56.474060 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 15:05:56.474522 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":86}
I20260619 15:05:56.478437 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.462934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48530 (local address 127.24.103.125:40973)
0619 15:05:56.463130 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:56.463135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:56.463151 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:05:56.463201 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:56.463204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:56.463254 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:56.463347 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:56.464523 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.465314 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.466517 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.466989 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.467125 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.472026 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:56.472033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:56.472044 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:56.472090 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:56.475608 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.476401 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.476407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.476411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.476605 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.476981 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.476987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.476991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.477292 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:56.477590 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:56.477854 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 15:05:56.478101 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 15:05:56.480742 6186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6: Bootstrap starting.
I20260619 15:05:56.483438 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:05:56.484536 6186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6: Neither blocks nor log segments found. Creating new log.
I20260619 15:05:56.485648 6186 log.cc:824] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6: Log is configured to *not* fsync() on all Append() calls
I20260619 15:05:56.489723 6186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6: No bootstrap required, opened a new log
I20260619 15:05:56.505178 6186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } }
I20260619 15:05:56.505806 6186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:05:56.505978 6186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 340bf1664d0b485692237518e295bcd6, State: Initialized, Role: FOLLOWER
I20260619 15:05:56.506804 6186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } }
I20260619 15:05:56.515216 6200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } } }
I20260619 15:05:56.515873 6200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:05:56.516846 6186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:05:56.522558 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.490684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.24.103.124:44139)
0619 15:05:56.490946 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:56.490954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:56.500915 (+ 9961us) server_negotiation.cc:407] Connection header received
0619 15:05:56.501220 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:56.501226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:56.501309 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:56.501389 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:56.504236 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.505486 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.508034 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.508735 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.508830 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.513682 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:56.513690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:56.513695 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:56.513739 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:56.518239 (+ 4500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.520110 (+ 1871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.520122 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.520125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.520319 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.520768 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.520776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.520780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.521141 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:56.521393 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:56.521922 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 15:05:56.522169 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 15:05:56.522800 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.491136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48538 (local address 127.24.103.125:40973)
0619 15:05:56.491326 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:56.491332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:56.503460 (+ 12128us) server_negotiation.cc:407] Connection header received
0619 15:05:56.503753 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:56.503760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:56.503842 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:56.503921 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:56.506121 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.507278 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.509439 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.510130 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.510242 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.513962 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:56.513968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:56.513972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:56.514020 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:56.518346 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.519639 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.519646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.519649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.519817 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.520275 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.520280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.520283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.520707 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:56.520949 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:56.522161 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0619 15:05:56.522408 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 15:05:56.525601 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.490178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.24.103.126:46673)
0619 15:05:56.490408 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:56.490417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:56.503050 (+ 12633us) server_negotiation.cc:407] Connection header received
0619 15:05:56.503513 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:56.503520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:56.503620 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:56.503717 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:56.506105 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.507222 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.510560 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.511568 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.511709 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.516860 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:56.516870 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:56.516876 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:56.516924 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:56.521268 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.522461 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.522470 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.522477 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.522679 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.523304 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.523312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.523318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.523789 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:56.524093 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:56.524833 (+ 740us) server_negotiation.cc:299] Negotiation successful
0619 15:05:56.525139 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:05:56.527654 6038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } attempt: 4
W20260619 15:05:56.535535 6211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 15:05:56.535980 6200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:05:56.536381 6200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } }
Jun 19 15:05:56 dist-test-slave-zpbw krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
W20260619 15:05:56.537034 6211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:05:56.538010 6200 leader_election.cc:288] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f37b193d85da4b90b249aefa7562e6b5 (127.24.103.126:46673), 88bd07e8bfbc48aa8c2845cd66d9220f (127.24.103.125:40973)
I20260619 15:05:56.545105 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.528256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:41134)
0619 15:05:56.528462 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.528488 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:56.528632 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:56.528951 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:56.528956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:56.529364 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:56.529519 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:56.529535 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.530893 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.530899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:56.532369 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:56.532388 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.533702 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.533710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:56.533896 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.534611 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:56.534635 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:56.537503 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:56.541934 (+ 4431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:56.541940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:56.541946 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:56.542516 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:56.543062 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:56.543067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:56.543070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:56.543244 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:56.544009 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:56.544014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:56.544199 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 15:05:56.544351 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 15:05:56.544663 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 15:05:56.546263 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.528317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.24.103.124:44139)
0619 15:05:56.528464 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:56.528468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:56.528549 (+ 81us) server_negotiation.cc:407] Connection header received
0619 15:05:56.528751 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:56.528755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:56.528812 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:56.528926 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:56.529661 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.530698 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.532531 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.533330 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.533463 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.537726 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:56.537732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:56.537736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:56.537773 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:56.541755 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.542710 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.542717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.542723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.542918 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.543390 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.543396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.543399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.543870 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:56.545482 (+ 1612us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:56.545615 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 15:05:56.545883 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 15:05:56.532070 6119 master_runner.cc:186] Error getting master registration for 127.24.103.126:46673: OK, instance_id { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" instance_seqno: 1781881555383274 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 15:05:56.549944 6119 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 15:05:56.552593 6038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5: Bootstrap starting.
I20260619 15:05:56.557132 6038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5: Neither blocks nor log segments found. Creating new log.
I20260619 15:05:56.558590 6038 log.cc:824] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5: Log is configured to *not* fsync() on all Append() calls
I20260619 15:05:56.562143 6038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5: No bootstrap required, opened a new log
I20260619 15:05:56.573685 6038 raft_consensus.cc:348] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } }
I20260619 15:05:56.574134 6038 raft_consensus.cc:374] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:05:56.574268 6038 raft_consensus.cc:729] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37b193d85da4b90b249aefa7562e6b5, State: Initialized, Role: FOLLOWER
I20260619 15:05:56.574990 6038 consensus_queue.cc:260] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } }
I20260619 15:05:56.576069 6214 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } } }
I20260619 15:05:56.576509 6214 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:05:56.577585 6038 sys_catalog.cc:564] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:05:56.577667 6013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "340bf1664d0b485692237518e295bcd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37b193d85da4b90b249aefa7562e6b5" is_pre_election: true
I20260619 15:05:56.578238 6013 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 340bf1664d0b485692237518e295bcd6 in term 0.
I20260619 15:05:56.579005 6135 leader_election.cc:302] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340bf1664d0b485692237518e295bcd6, f37b193d85da4b90b249aefa7562e6b5; no voters:
I20260619 15:05:56.579694 6200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:05:56.579861 6200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:05:56.579977 6200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:05:56.582849 6200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } }
I20260619 15:05:56.584313 6013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "340bf1664d0b485692237518e295bcd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37b193d85da4b90b249aefa7562e6b5"
I20260619 15:05:56.584573 6013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:05:56.585852 6200 leader_election.cc:288] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [CANDIDATE]: Term 1 election: Requested vote from peers f37b193d85da4b90b249aefa7562e6b5 (127.24.103.126:46673), 88bd07e8bfbc48aa8c2845cd66d9220f (127.24.103.125:40973)
I20260619 15:05:56.588351 6013 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 340bf1664d0b485692237518e295bcd6 in term 1.
I20260619 15:05:56.588807 6135 leader_election.cc:302] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340bf1664d0b485692237518e295bcd6, f37b193d85da4b90b249aefa7562e6b5; no voters:
I20260619 15:05:56.589128 6200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:05:56.589771 6200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [term 1 LEADER]: Becoming Leader. State: Replica: 340bf1664d0b485692237518e295bcd6, State: Running, Role: LEADER
I20260619 15:05:56.590384 6200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } }
I20260619 15:05:56.595885 6219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 340bf1664d0b485692237518e295bcd6. Latest consensus state: current_term: 1 leader_uuid: "340bf1664d0b485692237518e295bcd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } } }
I20260619 15:05:56.596278 6219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [sys.catalog]: This master's current role is: LEADER
I20260619 15:05:56.601132 6222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:05:56.606528 6222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:05:56.609786 6112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } attempt: 5
Jun 19 15:05:56 dist-test-slave-zpbw krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
W20260619 15:05:56.621765 5992 tablet.cc:2369] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:05:56.640561 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.612286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.24.103.124:44139)
0619 15:05:56.612529 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 15:05:56.612537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:05:56.612561 (+ 24us) server_negotiation.cc:407] Connection header received
0619 15:05:56.612632 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:05:56.612640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:05:56.612745 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:05:56.612880 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:05:56.613915 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.615190 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.617100 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.617805 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.617927 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.624279 (+ 6352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:05:56.624287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:05:56.624292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:05:56.624336 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:05:56.635312 (+ 10976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.636838 (+ 1526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.636860 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.636865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.637037 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:05:56.637781 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:05:56.637787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:05:56.637791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:05:56.638242 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:05:56.638405 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 15:05:56.639325 (+ 920us) server_negotiation.cc:299] Negotiation successful
0619 15:05:56.639833 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 15:05:56.642588 6140 tablet.cc:2369] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:05:56.643518 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:05:56.611308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44139 (local address 127.0.0.1:41138)
0619 15:05:56.611664 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 15:05:56.611687 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:05:56.611806 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:05:56.612922 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:05:56.612927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:05:56.613514 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:05:56.613741 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:56.613754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.615406 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.615412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:56.616889 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:05:56.616905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:05:56.618227 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:05:56.618236 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:05:56.618471 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:05:56.619212 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:05:56.619238 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:05:56.623952 (+ 4714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:05:56.635668 (+ 11716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:56.635676 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:56.635681 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:56.636603 (+ 922us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:56.637243 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:05:56.637269 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:05:56.637273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:05:56.637578 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:05:56.638587 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:05:56.638593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:05:56.638885 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 15:05:56.641256 (+ 2371us) client_negotiation.cc:240] Negotiation successful
0619 15:05:56.641821 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260619 15:05:56.649580 6013 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [term 1 FOLLOWER]: Refusing update from remote peer 340bf1664d0b485692237518e295bcd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:05:56.651037 6219 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:05:56.651713 6229 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 15:05:56.654132 6112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f: Bootstrap starting.
I20260619 15:05:56.661103 6112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f: Neither blocks nor log segments found. Creating new log.
I20260619 15:05:56.663205 6112 log.cc:824] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f: Log is configured to *not* fsync() on all Append() calls
I20260619 15:05:56.670922 6214 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 340bf1664d0b485692237518e295bcd6. Latest consensus state: current_term: 1 leader_uuid: "340bf1664d0b485692237518e295bcd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } } }
I20260619 15:05:56.671445 6214 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:05:56.673276 6112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f: No bootstrap required, opened a new log
I20260619 15:05:56.672652 5971 master_runner.cc:186] Error getting master registration for 127.24.103.125:40973: OK, instance_id { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" instance_seqno: 1781881555880711 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 15:05:56.684588 5971 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 15:05:56.685282 6112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } }
I20260619 15:05:56.685743 6112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:05:56.685870 6112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88bd07e8bfbc48aa8c2845cd66d9220f, State: Initialized, Role: FOLLOWER
I20260619 15:05:56.686483 6225 mvcc.cc:205] Tried to move back new op lower bound from 7298586856017383424 to 7298586855799316480. Current Snapshot: MvccSnapshot[applied={T|T < 7298586856017383424}]
I20260619 15:05:56.688678 6219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "340bf1664d0b485692237518e295bcd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } } }
I20260619 15:05:56.689239 6219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [sys.catalog]: This master's current role is: LEADER
I20260619 15:05:56.689567 6112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } }
I20260619 15:05:56.695010 6222 catalog_manager.cc:1347] Generated new cluster ID: 695878d45176457b8fbcdb965f9ebe78
I20260619 15:05:56.695147 6222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:05:56.707829 6214 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "340bf1664d0b485692237518e295bcd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } } }
I20260619 15:05:56.708137 6214 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:05:56.713577 6234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } } }
I20260619 15:05:56.715693 6222 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:05:56.717244 6222 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:05:56.715165 6087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "340bf1664d0b485692237518e295bcd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" is_pre_election: true
I20260619 15:05:56.719144 6234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:05:56.715507 6112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:05:56.715586 6085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:05:56.715776 6086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "340bf1664d0b485692237518e295bcd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f"
I20260619 15:05:56.723996 6085 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [term 1 FOLLOWER]: Refusing update from remote peer 340bf1664d0b485692237518e295bcd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:05:56.724942 6219 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:05:56.734193 6222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6: Generated new TSK 0
I20260619 15:05:56.735495 6222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:05:56.773699 6200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "340bf1664d0b485692237518e295bcd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } } }
I20260619 15:05:56.773651 6247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "340bf1664d0b485692237518e295bcd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } } }
I20260619 15:05:56.774132 6200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340bf1664d0b485692237518e295bcd6 [sys.catalog]: This master's current role is: LEADER
I20260619 15:05:56.774245 6247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:05:56.774559 6234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 340bf1664d0b485692237518e295bcd6. Latest consensus state: current_term: 1 leader_uuid: "340bf1664d0b485692237518e295bcd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37b193d85da4b90b249aefa7562e6b5" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46673 } } peers { permanent_uuid: "88bd07e8bfbc48aa8c2845cd66d9220f" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 40973 } } peers { permanent_uuid: "340bf1664d0b485692237518e295bcd6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44139 } } }
I20260619 15:05:56.774961 6234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f [sys.catalog]: This master's current role is: FOLLOWER
W20260619 15:05:56.789184 6254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:05:56.789472 6254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:05:57.656386 6229 catalog_manager.cc:1259] Loaded cluster ID: 695878d45176457b8fbcdb965f9ebe78
I20260619 15:05:57.656540 6229 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5: loading cluster ID for follower catalog manager: success
I20260619 15:05:57.659461 6229 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5: acquiring CA information for follower catalog manager: success
I20260619 15:05:57.661427 6229 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f37b193d85da4b90b249aefa7562e6b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 15:05:57.791373 6254 catalog_manager.cc:1259] Loaded cluster ID: 695878d45176457b8fbcdb965f9ebe78
I20260619 15:05:57.791518 6254 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f: loading cluster ID for follower catalog manager: success
I20260619 15:05:57.793984 6254 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f: acquiring CA information for follower catalog manager: success
I20260619 15:05:57.795935 6254 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88bd07e8bfbc48aa8c2845cd66d9220f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 15:05:57.858438 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 5971
I20260619 15:05:57.859819 6033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:05:57.973416 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 5971
I20260619 15:05:57.992408 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 6045
I20260619 15:05:57.993577 6107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:05:58.101540 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 6045
I20260619 15:05:58.121428 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 6119
I20260619 15:05:58.122534 6181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:05:58.231012 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 6119
2026-06-19T15:05:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4607 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:05:58 dist-test-slave-zpbw krb5kdc[6262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:05:58 dist-test-slave-zpbw krb5kdc[6262](info): set up 2 sockets
Jun 19 15:05:58 dist-test-slave-zpbw krb5kdc[6262](info): commencing operation
krb5kdc: starting...
W20260619 15:05:59.347563 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:05:59 dist-test-slave-zpbw krb5kdc[6262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:05:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:05:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:05:59.592396 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43381
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:43381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:05:59.930233 6275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:05:59.930634 6275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:05:59.930830 6275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:05:59.938328 6275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:05:59.938505 6275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:05:59.938608 6275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:05:59.938699 6275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:05:59.938805 6275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:05:59.948997 6275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:43381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:05:59.950446 6275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:05:59.951839 6275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:05:59.964370 6283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:59.964406 6280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:05:59.964540 6281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:05:59.965004 6275 server_base.cc:1034] running on GCE node
I20260619 15:05:59.965881 6275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:05:59.967483 6275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:05:59.968719 6275 hybrid_clock.cc:648] HybridClock initialized: now 1781881559968662 us; error 49 us; skew 500 ppm
I20260619 15:05:59.969079 6275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:05:59 dist-test-slave-zpbw krb5kdc[6262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:05:59.972414 6275 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:05:59.974090 6275 webserver.cc:460] Webserver started at http://127.24.103.126:41467/ using document root <none> and password file <none>
I20260619 15:05:59.974911 6275 fs_manager.cc:359] Metadata directory not provided
I20260619 15:05:59.975060 6275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:05:59.975404 6275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:05:59.978593 6275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "1c75efcfb1364f9c9b36aa101a5a6b6e"
format_stamp: "Formatted at 2026-06-19 15:05:59 on dist-test-slave-zpbw"
server_key: "d425d6282aa0c5c636527c2a5a620be7"
server_key_iv: "6e9e0c9ba50767cd42ed6d745f119b6a"
server_key_version: "encryptionkey@0"
I20260619 15:05:59.979573 6275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "1c75efcfb1364f9c9b36aa101a5a6b6e"
format_stamp: "Formatted at 2026-06-19 15:05:59 on dist-test-slave-zpbw"
server_key: "d425d6282aa0c5c636527c2a5a620be7"
server_key_iv: "6e9e0c9ba50767cd42ed6d745f119b6a"
server_key_version: "encryptionkey@0"
I20260619 15:05:59.985386 6275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:05:59.989216 6290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:05:59.990767 6275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:05:59.991036 6275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "1c75efcfb1364f9c9b36aa101a5a6b6e"
format_stamp: "Formatted at 2026-06-19 15:05:59 on dist-test-slave-zpbw"
server_key: "d425d6282aa0c5c636527c2a5a620be7"
server_key_iv: "6e9e0c9ba50767cd42ed6d745f119b6a"
server_key_version: "encryptionkey@0"
I20260619 15:05:59.991263 6275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:00.016161 6275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:00.020838 6275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:00.021157 6275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:00.040732 6275 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:43381
I20260619 15:06:00.040746 6341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:43381 every 8 connection(s)
I20260619 15:06:00.042213 6275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:06:00.046233 6342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:00.046438 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 6275
I20260619 15:06:00.046682 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:06:00.047397 24989 external_mini_cluster.cc:1428] Setting key fe0ffc02008aefec1c785600704821cd
I20260619 15:06:00.061527 6342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e: Bootstrap starting.
Jun 19 15:06:00 dist-test-slave-zpbw krb5kdc[6262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:00.065177 6342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:00.066435 6342 log.cc:824] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:00.069772 6342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e: No bootstrap required, opened a new log
I20260619 15:06:00.073841 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:00.049378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.24.103.126:43381)
0619 15:06:00.050220 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:00.050235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:00.050287 (+ 52us) server_negotiation.cc:407] Connection header received
0619 15:06:00.053289 (+ 3002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:00.053326 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:00.053954 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:00.054465 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:00.056070 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:00.058127 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:00.059611 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:00.060720 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:00.060925 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:00.065836 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:00.065845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:00.065861 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:00.065909 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:00.069160 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:00.070136 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:00.070143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:00.070150 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:00.070350 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:00.070884 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:00.070887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:00.070889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:00.071468 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:00.071714 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:00.072268 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 15:06:00.072723 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":352,"threads_started":1}
I20260619 15:06:00.077968 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:06:00.078979 6342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c75efcfb1364f9c9b36aa101a5a6b6e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43381 } }
I20260619 15:06:00.079411 6342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:00.079505 6342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c75efcfb1364f9c9b36aa101a5a6b6e, State: Initialized, Role: FOLLOWER
I20260619 15:06:00.080101 6342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c75efcfb1364f9c9b36aa101a5a6b6e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43381 } }
I20260619 15:06:00.080334 6342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:06:00.080498 6342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:06:00.080705 6342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:00.083244 6342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c75efcfb1364f9c9b36aa101a5a6b6e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43381 } }
I20260619 15:06:00.083781 6342 leader_election.cc:302] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c75efcfb1364f9c9b36aa101a5a6b6e; no voters:
I20260619 15:06:00.084380 6342 leader_election.cc:288] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:06:00.084517 6347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:06:00.085476 6347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [term 1 LEADER]: Becoming Leader. State: Replica: 1c75efcfb1364f9c9b36aa101a5a6b6e, State: Running, Role: LEADER
I20260619 15:06:00.086009 6347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c75efcfb1364f9c9b36aa101a5a6b6e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43381 } }
I20260619 15:06:00.086683 6342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:06:00.088966 6349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c75efcfb1364f9c9b36aa101a5a6b6e. Latest consensus state: current_term: 1 leader_uuid: "1c75efcfb1364f9c9b36aa101a5a6b6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c75efcfb1364f9c9b36aa101a5a6b6e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43381 } } }
I20260619 15:06:00.089462 6349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [sys.catalog]: This master's current role is: LEADER
I20260619 15:06:00.091511 6348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c75efcfb1364f9c9b36aa101a5a6b6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c75efcfb1364f9c9b36aa101a5a6b6e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43381 } } }
I20260619 15:06:00.091791 6348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e [sys.catalog]: This master's current role is: LEADER
I20260619 15:06:00.092097 6354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:06:00.098922 6354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:06:00.108467 6354 catalog_manager.cc:1347] Generated new cluster ID: 60a82639bd0f4a29bb31409db0effc79
I20260619 15:06:00.108604 6354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:06:00.125015 6354 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:06:00.127063 6354 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:06:00.135272 6354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c75efcfb1364f9c9b36aa101a5a6b6e: Generated new TSK 0
I20260619 15:06:00.136492 6354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:06:00.190994 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 6275
I20260619 15:06:00.192307 6337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:00.289175 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 6275
2026-06-19T15:06:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2052 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:06:00 dist-test-slave-zpbw krb5kdc[6370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:06:00 dist-test-slave-zpbw krb5kdc[6370](info): set up 2 sockets
Jun 19 15:06:00 dist-test-slave-zpbw krb5kdc[6370](info): commencing operation
krb5kdc: starting...
W20260619 15:06:01.404023 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:06:01 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:06:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:06:01.643718 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44191
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:44191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:01.994565 6383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:01.995009 6383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:01.995199 6383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:02.003005 6383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:06:02.003144 6383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:02.003247 6383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:02.003346 6383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:06:02.003427 6383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:06:02.013870 6383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:44191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:02.015444 6383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:02.016829 6383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:02.029387 6391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:02.029420 6389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:02.029475 6388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:02.030447 6383 server_base.cc:1034] running on GCE node
I20260619 15:06:02.031411 6383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:02.033001 6383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:02.034237 6383 hybrid_clock.cc:648] HybridClock initialized: now 1781881562034182 us; error 46 us; skew 500 ppm
I20260619 15:06:02.034626 6383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:02 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881562, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:02.041435 6383 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:06:02.043340 6383 webserver.cc:460] Webserver started at http://127.24.103.126:34617/ using document root <none> and password file <none>
I20260619 15:06:02.044117 6383 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:02.044231 6383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:02.044502 6383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:06:02.047789 6383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "e618e0c94f5c40e7a1b783f661673c08"
format_stamp: "Formatted at 2026-06-19 15:06:02 on dist-test-slave-zpbw"
server_key: "a4a9f86fc96168efb8e4620a2161ff36"
server_key_iv: "9e26b9d10b2cf8dc4ee83ebda0634eba"
server_key_version: "encryptionkey@0"
I20260619 15:06:02.048574 6383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "e618e0c94f5c40e7a1b783f661673c08"
format_stamp: "Formatted at 2026-06-19 15:06:02 on dist-test-slave-zpbw"
server_key: "a4a9f86fc96168efb8e4620a2161ff36"
server_key_iv: "9e26b9d10b2cf8dc4ee83ebda0634eba"
server_key_version: "encryptionkey@0"
I20260619 15:06:02.054164 6383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:06:02.058063 6398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:02.059665 6383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:06:02.059885 6383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "e618e0c94f5c40e7a1b783f661673c08"
format_stamp: "Formatted at 2026-06-19 15:06:02 on dist-test-slave-zpbw"
server_key: "a4a9f86fc96168efb8e4620a2161ff36"
server_key_iv: "9e26b9d10b2cf8dc4ee83ebda0634eba"
server_key_version: "encryptionkey@0"
I20260619 15:06:02.060118 6383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:02.079572 6383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:02.084180 6383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:02.084519 6383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:02.103727 6383 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:44191
I20260619 15:06:02.103765 6449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:44191 every 8 connection(s)
I20260619 15:06:02.105314 6383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:06:02.107817 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 6383
I20260619 15:06:02.108058 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:06:02.108743 24989 external_mini_cluster.cc:1428] Setting key 8e83d245e34b42c592ce48200b4bd51c
I20260619 15:06:02.109901 6450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 15:06:02 dist-test-slave-zpbw krb5kdc[6370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:02.125067 6450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Bootstrap starting.
I20260619 15:06:02.129614 6450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:02.131069 6450 log.cc:824] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:02.134291 6450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: No bootstrap required, opened a new log
I20260619 15:06:02.135187 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:02.110938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.24.103.126:44191)
0619 15:06:02.111838 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:02.111857 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:02.111912 (+ 55us) server_negotiation.cc:407] Connection header received
0619 15:06:02.113818 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:02.113850 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:02.114452 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:02.114958 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:02.116514 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:02.118648 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:02.120192 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:02.121341 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:02.121564 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:02.126062 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:02.126073 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:02.126094 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:02.126145 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:02.130402 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:02.131378 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:02.131386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:02.131393 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:02.131596 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:02.132067 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:02.132071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:02.132073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:02.132674 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:02.133031 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:02.133492 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 15:06:02.133961 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":371,"threads_started":1}
I20260619 15:06:02.142930 6450 raft_consensus.cc:348] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } }
I20260619 15:06:02.143412 6450 raft_consensus.cc:374] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:02.143527 6450 raft_consensus.cc:729] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e618e0c94f5c40e7a1b783f661673c08, State: Initialized, Role: FOLLOWER
I20260619 15:06:02.144142 6450 consensus_queue.cc:260] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } }
I20260619 15:06:02.144400 6450 raft_consensus.cc:388] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:06:02.144544 6450 raft_consensus.cc:482] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:06:02.144722 6450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:02.147284 6450 raft_consensus.cc:504] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } }
I20260619 15:06:02.147857 6450 leader_election.cc:302] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e618e0c94f5c40e7a1b783f661673c08; no voters:
I20260619 15:06:02.148463 6450 leader_election.cc:288] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:06:02.148625 6455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:06:02.149785 6455 raft_consensus.cc:686] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 1 LEADER]: Becoming Leader. State: Replica: e618e0c94f5c40e7a1b783f661673c08, State: Running, Role: LEADER
I20260619 15:06:02.150456 6455 consensus_queue.cc:237] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } }
I20260619 15:06:02.150885 6450 sys_catalog.cc:564] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:06:02.153725 6456 sys_catalog.cc:455] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e618e0c94f5c40e7a1b783f661673c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } } }
I20260619 15:06:02.154258 6456 sys_catalog.cc:458] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: This master's current role is: LEADER
I20260619 15:06:02.153672 6457 sys_catalog.cc:455] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e618e0c94f5c40e7a1b783f661673c08. Latest consensus state: current_term: 1 leader_uuid: "e618e0c94f5c40e7a1b783f661673c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } } }
I20260619 15:06:02.154839 6457 sys_catalog.cc:458] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: This master's current role is: LEADER
I20260619 15:06:02.155407 6464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:06:02.170061 6464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:06:02.179347 6464 catalog_manager.cc:1347] Generated new cluster ID: 5e415f024e384b7781b300a54b953d70
I20260619 15:06:02.179492 6464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:06:02.199528 6464 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:06:02.201546 6464 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:06:02.213163 6464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Generated new TSK 0
I20260619 15:06:02.214325 6464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:06:02.286362 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44191
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:02.638589 6476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:02.639016 6476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:02.639191 6476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:02.646850 6476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:02.647039 6476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:02.647229 6476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:06:02.657821 6476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:02.659314 6476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:02.660782 6476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:02.674551 6481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:02.674573 6482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:02.674734 6484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:02.675320 6476 server_base.cc:1034] running on GCE node
I20260619 15:06:02.676090 6476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:02.677115 6476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:02.678355 6476 hybrid_clock.cc:648] HybridClock initialized: now 1781881562678286 us; error 57 us; skew 500 ppm
I20260619 15:06:02.678694 6476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:02 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881562, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:02.682451 6476 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:06:02.684121 6476 webserver.cc:460] Webserver started at http://127.24.103.65:34843/ using document root <none> and password file <none>
I20260619 15:06:02.684890 6476 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:02.685045 6476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:02.685366 6476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:06:02.688583 6476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "9f573cf213df46329ff2000e66667ad0"
format_stamp: "Formatted at 2026-06-19 15:06:02 on dist-test-slave-zpbw"
server_key: "39f2da34ee05705cb0146d6ca094e2ba"
server_key_iv: "c2ab20bd0e6a25eddde8726c7fd7a6d8"
server_key_version: "encryptionkey@0"
I20260619 15:06:02.689410 6476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "9f573cf213df46329ff2000e66667ad0"
format_stamp: "Formatted at 2026-06-19 15:06:02 on dist-test-slave-zpbw"
server_key: "39f2da34ee05705cb0146d6ca094e2ba"
server_key_iv: "c2ab20bd0e6a25eddde8726c7fd7a6d8"
server_key_version: "encryptionkey@0"
I20260619 15:06:02.695145 6476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:06:02.698841 6491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:02.700269 6476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:06:02.700469 6476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "9f573cf213df46329ff2000e66667ad0"
format_stamp: "Formatted at 2026-06-19 15:06:02 on dist-test-slave-zpbw"
server_key: "39f2da34ee05705cb0146d6ca094e2ba"
server_key_iv: "c2ab20bd0e6a25eddde8726c7fd7a6d8"
server_key_version: "encryptionkey@0"
I20260619 15:06:02.700683 6476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:02.724289 6476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:02.728780 6476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:02.729107 6476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:02.730152 6476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:02.731724 6476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:02.731837 6476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:02.732002 6476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:02.732077 6476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:02.767788 6476 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:40079
I20260619 15:06:02.767839 6603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:40079 every 8 connection(s)
I20260619 15:06:02.769265 6476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:06:02.773265 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 6476
I20260619 15:06:02.773538 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:06:02.774437 24989 external_mini_cluster.cc:1428] Setting key 13d8f01ec42f5a769a3e47468abec890
Jun 19 15:06:02 dist-test-slave-zpbw krb5kdc[6370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881562, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:02.794569 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:02.772142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:50015 (local address 127.24.103.126:44191)
0619 15:06:02.772633 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:02.772639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:02.773904 (+ 1265us) server_negotiation.cc:407] Connection header received
0619 15:06:02.776255 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:02.776261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:02.776345 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:02.776483 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:02.778742 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:02.779539 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:02.782289 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:02.782710 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:02.782886 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:02.787271 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:02.787281 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:02.787289 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:02.787354 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:02.790965 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:02.791793 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:02.791802 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:02.791808 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:02.791973 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:02.792415 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:02.792423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:02.792429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:02.792856 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:02.793004 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:02.793765 (+ 761us) server_negotiation.cc:299] Negotiation successful
0619 15:06:02.794055 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":241,"threads_started":1}
I20260619 15:06:02.795313 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:02.772782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:50015)
0619 15:06:02.773696 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:02.773760 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:02.775969 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:02.776768 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:02.776780 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:02.777602 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:06:02.778525 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:02.778557 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:02.779731 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:02.779740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:02.782109 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:02.782132 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:02.783016 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:02.783027 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:02.783389 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:02.784199 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:06:02.784236 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:06:02.787006 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:06:02.791132 (+ 4126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:02.791140 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:02.791156 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:02.791650 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:02.792096 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:02.792099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:02.792102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:02.792297 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:02.792999 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:06:02.793006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:06:02.793358 (+ 352us) client_negotiation.cc:769] Sending connection context
0619 15:06:02.793618 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 15:06:02.794084 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":340,"threads_started":1}
I20260619 15:06:02.797103 6604 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44191
I20260619 15:06:02.797488 6604 heartbeater.cc:464] Registering TS with master...
I20260619 15:06:02.798502 6604 heartbeater.cc:511] Master 127.24.103.126:44191 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260619 15:06:02.801082 6415 ts_manager.cc:194] Registered new tserver with Master: 9f573cf213df46329ff2000e66667ad0 (127.24.103.65:40079)
I20260619 15:06:02.802728 6415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:50015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:06:02.831280 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44191
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:03.186110 6610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:03.186510 6610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:03.186738 6610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:03.194525 6610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:03.194698 6610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:03.194921 6610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:06:03.205893 6610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:03.207460 6610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:03.208942 6610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:03.223351 6615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:03.223559 6618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:03.223582 6616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:03.228402 6610 server_base.cc:1034] running on GCE node
I20260619 15:06:03.229115 6610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:03.230163 6610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:03.231392 6610 hybrid_clock.cc:648] HybridClock initialized: now 1781881563231333 us; error 67 us; skew 500 ppm
I20260619 15:06:03.231724 6610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:03 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881563, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:03.235159 6610 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:06:03.236817 6610 webserver.cc:460] Webserver started at http://127.24.103.66:35005/ using document root <none> and password file <none>
I20260619 15:06:03.237583 6610 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:03.237733 6610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:03.238049 6610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:06:03.241192 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "d416e496633a4b4ca187f7b0537adcca"
format_stamp: "Formatted at 2026-06-19 15:06:03 on dist-test-slave-zpbw"
server_key: "d227d62a65da6cfa1175ea6a2df88d36"
server_key_iv: "8860e2e2ce256bf49bb5a6ef75de022a"
server_key_version: "encryptionkey@0"
I20260619 15:06:03.242044 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "d416e496633a4b4ca187f7b0537adcca"
format_stamp: "Formatted at 2026-06-19 15:06:03 on dist-test-slave-zpbw"
server_key: "d227d62a65da6cfa1175ea6a2df88d36"
server_key_iv: "8860e2e2ce256bf49bb5a6ef75de022a"
server_key_version: "encryptionkey@0"
I20260619 15:06:03.247745 6610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 15:06:03.251659 6625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:03.253214 6610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:06:03.253410 6610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "d416e496633a4b4ca187f7b0537adcca"
format_stamp: "Formatted at 2026-06-19 15:06:03 on dist-test-slave-zpbw"
server_key: "d227d62a65da6cfa1175ea6a2df88d36"
server_key_iv: "8860e2e2ce256bf49bb5a6ef75de022a"
server_key_version: "encryptionkey@0"
I20260619 15:06:03.253648 6610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:03.271479 6610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:03.275985 6610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:03.276297 6610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:03.277295 6610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:03.278748 6610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:03.278906 6610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 15:06:03.279066 6610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:03.279139 6610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:03.313823 6610 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:35171
I20260619 15:06:03.313879 6737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:35171 every 8 connection(s)
I20260619 15:06:03.315277 6610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:06:03.316845 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 6610
I20260619 15:06:03.317039 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:06:03.317497 24989 external_mini_cluster.cc:1428] Setting key f80dfc004ff046d03b5fc04007d2a71c
Jun 19 15:06:03 dist-test-slave-zpbw krb5kdc[6370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881563, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:03.339401 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:03.318024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:38121 (local address 127.24.103.126:44191)
0619 15:06:03.319175 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:03.319181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:03.319857 (+ 676us) server_negotiation.cc:407] Connection header received
0619 15:06:03.321757 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:03.321765 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:03.321856 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:03.321974 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:03.323600 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.324854 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.327105 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.327596 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.327760 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:03.332659 (+ 4899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:03.332665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:03.332669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:03.332707 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:03.335551 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:03.336674 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:03.336678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:03.336680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:03.336817 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:03.337381 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:03.337384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:03.337386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:03.337664 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:03.337776 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:03.338861 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0619 15:06:03.339058 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":900,"threads_started":1}
I20260619 15:06:03.341156 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:03.318739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:38121)
0619 15:06:03.319696 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:03.319741 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:03.321478 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:03.322203 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:03.322211 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:03.322757 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:06:03.323418 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:03.323436 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.325012 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.325019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:03.326951 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:03.326965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.327855 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.327863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:03.328243 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:03.329051 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:06:03.329086 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:06:03.332324 (+ 3238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:06:03.335739 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:03.335753 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:03.335775 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:03.336532 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:03.336949 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:03.336957 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:03.336964 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:03.337257 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:03.337796 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:06:03.337808 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:06:03.338362 (+ 554us) client_negotiation.cc:769] Sending connection context
0619 15:06:03.338712 (+ 350us) client_negotiation.cc:240] Negotiation successful
0619 15:06:03.339467 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":388,"threads_started":1}
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260619 15:06:03.343000 6738 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44191
I20260619 15:06:03.343348 6738 heartbeater.cc:464] Registering TS with master...
I20260619 15:06:03.344206 6738 heartbeater.cc:511] Master 127.24.103.126:44191 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
I20260619 15:06:03.345842 6415 ts_manager.cc:194] Registered new tserver with Master: d416e496633a4b4ca187f7b0537adcca (127.24.103.66:35171)
I20260619 15:06:03.347189 6415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:38121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:06:03.374414 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44191
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:03.711414 6744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:03.711776 6744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:03.711956 6744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:03.719702 6744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:03.719868 6744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:03.720064 6744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:06:03.730621 6744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:03.732043 6744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:03.733435 6744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:03.746927 6750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:03.746969 6752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:03.746927 6749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:03.747608 6744 server_base.cc:1034] running on GCE node
I20260619 15:06:03.748278 6744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:03.749284 6744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:03.750516 6744 hybrid_clock.cc:648] HybridClock initialized: now 1781881563750422 us; error 63 us; skew 500 ppm
I20260619 15:06:03.750947 6744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:03 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881563, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:03.754354 6744 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:06:03.755874 6744 webserver.cc:460] Webserver started at http://127.24.103.67:43317/ using document root <none> and password file <none>
I20260619 15:06:03.756713 6744 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:03.756850 6744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:03.757226 6744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:06:03.760536 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "f1aea936947e41709523a572b8836648"
format_stamp: "Formatted at 2026-06-19 15:06:03 on dist-test-slave-zpbw"
server_key: "14b75e4533e26ac3450a5cff77dc6dbb"
server_key_iv: "20d5b5ff8b38b3f1b32e4e145174970f"
server_key_version: "encryptionkey@0"
I20260619 15:06:03.761366 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "f1aea936947e41709523a572b8836648"
format_stamp: "Formatted at 2026-06-19 15:06:03 on dist-test-slave-zpbw"
server_key: "14b75e4533e26ac3450a5cff77dc6dbb"
server_key_iv: "20d5b5ff8b38b3f1b32e4e145174970f"
server_key_version: "encryptionkey@0"
I20260619 15:06:03.767273 6744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 15:06:03.771972 6759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:03.773666 6744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:06:03.773818 6744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "f1aea936947e41709523a572b8836648"
format_stamp: "Formatted at 2026-06-19 15:06:03 on dist-test-slave-zpbw"
server_key: "14b75e4533e26ac3450a5cff77dc6dbb"
server_key_iv: "20d5b5ff8b38b3f1b32e4e145174970f"
server_key_version: "encryptionkey@0"
I20260619 15:06:03.774017 6744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:03.794111 6744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:03.798400 6744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:03.798666 6744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:03.799692 6744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:03.800994 6744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:03.801079 6744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:03.801167 6744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:03.801210 6744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:03.806409 6604 heartbeater.cc:503] Master 127.24.103.126:44191 was elected leader, sending a full tablet report...
I20260619 15:06:03.835371 6744 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:41901
I20260619 15:06:03.835467 6871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:41901 every 8 connection(s)
I20260619 15:06:03.836728 6744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:06:03.838166 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 6744
I20260619 15:06:03.838392 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:06:03.838927 24989 external_mini_cluster.cc:1428] Setting key 3e9d746f19c840e96f2076d55df64791
Jun 19 15:06:03 dist-test-slave-zpbw krb5kdc[6370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881563, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:03.857928 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:03.839640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:51053 (local address 127.24.103.126:44191)
0619 15:06:03.839860 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:03.839866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:03.841414 (+ 1548us) server_negotiation.cc:407] Connection header received
0619 15:06:03.843552 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:03.843557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:03.843662 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:03.843769 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:03.845257 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.845963 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.847605 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.847991 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.848137 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:03.851598 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:03.851603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:03.851606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:03.851640 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:03.855088 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:03.855830 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:03.855834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:03.855837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:03.855954 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:03.856411 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:03.856416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:03.856419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:03.856686 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:03.856817 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:03.857516 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 15:06:03.857688 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 15:06:03.859016 6874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:03.840332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:51053)
0619 15:06:03.841242 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:03.841298 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:03.843290 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:03.844010 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:03.844021 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:03.844532 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:06:03.845090 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:03.845106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.846119 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.846129 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:03.847418 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:03.847429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.848228 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.848235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:03.848461 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:03.849125 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:06:03.849151 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:06:03.851392 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:06:03.855219 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:03.855232 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:03.855259 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:03.855719 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:03.856052 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:03.856055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:03.856057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:03.856236 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:03.856790 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:06:03.856795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:06:03.857152 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 15:06:03.857371 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 15:06:03.857817 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":348,"threads_started":1}
I20260619 15:06:03.860407 6872 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44191
I20260619 15:06:03.860728 6872 heartbeater.cc:464] Registering TS with master...
I20260619 15:06:03.861501 6872 heartbeater.cc:511] Master 127.24.103.126:44191 requested a full tablet report, sending...
I20260619 15:06:03.862890 6415 ts_manager.cc:194] Registered new tserver with Master: f1aea936947e41709523a572b8836648 (127.24.103.67:41901)
I20260619 15:06:03.863701 6415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:51053
I20260619 15:06:03.867523 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:06:03.885057 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:03.872027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.24.103.126:44191)
0619 15:06:03.872218 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:03.872222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:03.872603 (+ 381us) server_negotiation.cc:407] Connection header received
0619 15:06:03.872755 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:03.872759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:03.872821 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:03.872920 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:03.874274 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.875143 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.876554 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.877048 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.877254 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:03.879956 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:03.879961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:03.879964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:03.879998 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:03.882382 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:03.883123 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:03.883127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:03.883130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:03.883270 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:03.883603 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:03.883606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:03.883608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:03.883940 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:03.884130 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:03.884517 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 15:06:03.884717 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 15:06:03.889326 6415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:06:03.896972 6415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:06:03.922396 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:03.910952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:40079 (local address 127.0.0.1:47808)
0619 15:06:03.912312 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:03.912351 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:03.912508 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:03.913839 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:03.913843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:03.913865 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:03.914402 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:03.914416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.916635 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.916643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:03.919306 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:03.919322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.921197 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.921204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:03.921457 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:03.921520 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 15:06:03.921657 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 15:06:03.921885 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"mutex_wait_us":64,"thread_start_us":1071,"threads_started":1}
I20260619 15:06:03.922456 6884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:03.910291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:35171 (local address 127.0.0.1:50712)
0619 15:06:03.912312 (+ 2021us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:03.912351 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:03.912561 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:03.913641 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:03.913652 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:03.913691 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:03.914259 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:03.914275 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.916404 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.916409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:03.919080 (+ 2671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:03.919099 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.921498 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.921504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:03.921689 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:03.921722 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 15:06:03.921808 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 15:06:03.921942 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1746,"mutex_wait_us":120,"thread_start_us":1551,"threads_started":1}
I20260619 15:06:03.923166 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:03.910448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.24.103.66:35171)
0619 15:06:03.911793 (+ 1345us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:03.911807 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:03.912452 (+ 645us) server_negotiation.cc:407] Connection header received
0619 15:06:03.912810 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:03.912822 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:03.913100 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:03.913299 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:03.914508 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.916259 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.919308 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.920957 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.921117 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:03.921337 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:03.922618 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0619 15:06:03.922832 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155,"thread_start_us":1010,"threads_started":1}
I20260619 15:06:03.923499 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:03.911549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.24.103.65:40079)
0619 15:06:03.913258 (+ 1709us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:03.913266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:03.913293 (+ 27us) server_negotiation.cc:407] Connection header received
0619 15:06:03.913390 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:03.913399 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:03.913626 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:03.914686 (+ 1060us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:03.914949 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.916458 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.919593 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.920966 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.921134 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:03.921382 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:03.922899 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0619 15:06:03.923099 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1530,"thread_start_us":1386,"threads_started":1}
I20260619 15:06:03.925094 6888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:03.912744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:41901 (local address 127.0.0.1:45758)
0619 15:06:03.915812 (+ 3068us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:03.915827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:03.915974 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:03.916563 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:03.916567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:03.916599 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:03.917192 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:03.917204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.920370 (+ 3166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.920376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:03.922584 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:03.922599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.924451 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.924457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:03.924634 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:03.924672 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 15:06:03.924752 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 15:06:03.924851 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2956,"thread_start_us":2809,"threads_started":1}
I20260619 15:06:03.925339 6889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:03.913287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.24.103.67:41901)
0619 15:06:03.915346 (+ 2059us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:03.915358 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:03.915931 (+ 573us) server_negotiation.cc:407] Connection header received
0619 15:06:03.916140 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:03.916146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:03.916317 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:03.916548 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:03.918539 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.920203 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.922805 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:03.924225 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:03.924358 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:03.924611 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:03.924825 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 15:06:03.925003 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1859,"thread_start_us":1687,"threads_started":1}
I20260619 15:06:03.927893 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 384098af5dea42a58138c0eef02ee160 (DEFAULT_TABLE table=test-table [id=adf2a181dfac48a4a2b197d54e55831a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:06:03.928893 6807 tablet_service.cc:1467] Processing CreateTablet for tablet 384098af5dea42a58138c0eef02ee160 (DEFAULT_TABLE table=test-table [id=adf2a181dfac48a4a2b197d54e55831a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:06:03.929502 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 384098af5dea42a58138c0eef02ee160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:03.929133 6673 tablet_service.cc:1467] Processing CreateTablet for tablet 384098af5dea42a58138c0eef02ee160 (DEFAULT_TABLE table=test-table [id=adf2a181dfac48a4a2b197d54e55831a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:06:03.930351 6673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 384098af5dea42a58138c0eef02ee160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:03.930348 6807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 384098af5dea42a58138c0eef02ee160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:03.942804 6890 tablet_bootstrap.cc:492] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca: Bootstrap starting.
I20260619 15:06:03.945387 6892 tablet_bootstrap.cc:492] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648: Bootstrap starting.
I20260619 15:06:03.946646 6890 tablet_bootstrap.cc:654] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:03.948056 6890 log.cc:824] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:03.949206 6892 tablet_bootstrap.cc:654] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:03.950497 6892 log.cc:824] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:03.950632 6891 tablet_bootstrap.cc:492] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0: Bootstrap starting.
I20260619 15:06:03.951540 6890 tablet_bootstrap.cc:492] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca: No bootstrap required, opened a new log
I20260619 15:06:03.951855 6890 ts_tablet_manager.cc:1397] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 15:06:03.953356 6892 tablet_bootstrap.cc:492] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648: No bootstrap required, opened a new log
I20260619 15:06:03.953640 6892 ts_tablet_manager.cc:1397] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 15:06:03.954392 6891 tablet_bootstrap.cc:654] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:03.955747 6891 log.cc:824] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:03.959033 6891 tablet_bootstrap.cc:492] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0: No bootstrap required, opened a new log
I20260619 15:06:03.959313 6891 ts_tablet_manager.cc:1397] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260619 15:06:03.962417 6892 raft_consensus.cc:348] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } } peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } }
I20260619 15:06:03.962427 6890 raft_consensus.cc:348] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } } peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } }
I20260619 15:06:03.962829 6892 raft_consensus.cc:374] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:03.962922 6890 raft_consensus.cc:374] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:03.962956 6892 raft_consensus.cc:729] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1aea936947e41709523a572b8836648, State: Initialized, Role: FOLLOWER
I20260619 15:06:03.963039 6890 raft_consensus.cc:729] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d416e496633a4b4ca187f7b0537adcca, State: Initialized, Role: FOLLOWER
I20260619 15:06:03.963639 6892 consensus_queue.cc:260] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } } peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } }
I20260619 15:06:03.963716 6890 consensus_queue.cc:260] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } } peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } }
I20260619 15:06:03.964885 6872 heartbeater.cc:503] Master 127.24.103.126:44191 was elected leader, sending a full tablet report...
I20260619 15:06:03.966749 6892 ts_tablet_manager.cc:1428] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260619 15:06:03.967818 6738 heartbeater.cc:503] Master 127.24.103.126:44191 was elected leader, sending a full tablet report...
I20260619 15:06:03.968397 6890 ts_tablet_manager.cc:1428] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260619 15:06:03.971917 6891 raft_consensus.cc:348] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } } peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } }
I20260619 15:06:03.972364 6891 raft_consensus.cc:374] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:03.972502 6891 raft_consensus.cc:729] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f573cf213df46329ff2000e66667ad0, State: Initialized, Role: FOLLOWER
I20260619 15:06:03.973187 6891 consensus_queue.cc:260] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } } peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } }
I20260619 15:06:03.975098 6891 ts_tablet_manager.cc:1428] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
W20260619 15:06:04.021813 6605 tablet.cc:2369] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:06:04.067409 6739 tablet.cc:2369] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:06:04.088835 6873 tablet.cc:2369] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:06:04.117313 6898 raft_consensus.cc:482] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:06:04.117615 6898 raft_consensus.cc:504] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } } peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } }
I20260619 15:06:04.119377 6898 leader_election.cc:288] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d416e496633a4b4ca187f7b0537adcca (127.24.103.66:35171), f1aea936947e41709523a572b8836648 (127.24.103.67:41901)
I20260619 15:06:04.126458 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:04.119891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:39917 (local address 127.24.103.66:35171)
0619 15:06:04.120161 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:04.120173 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:04.120618 (+ 445us) server_negotiation.cc:407] Connection header received
0619 15:06:04.120844 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:04.120852 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:04.120944 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:04.121059 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:04.121638 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.122638 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.124616 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.125503 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.125697 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:04.125775 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:04.126061 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 15:06:04.126187 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 15:06:04.126581 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:04.119816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:35171 (local address 127.24.103.65:39917)
0619 15:06:04.120453 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:04.120470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:04.120669 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:04.121133 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:04.121140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:04.121173 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:04.121518 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:04.121526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.122879 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.122883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:04.124434 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:04.124447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.125709 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.125715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:04.125888 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:04.125917 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 15:06:04.126033 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 15:06:04.126172 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":380,"threads_started":1}
I20260619 15:06:04.126660 6889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:04.120810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:43875 (local address 127.24.103.67:41901)
0619 15:06:04.121042 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:04.121047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:04.121385 (+ 338us) server_negotiation.cc:407] Connection header received
0619 15:06:04.121582 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:04.121589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:04.121661 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:04.121753 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:04.122253 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.123177 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.124958 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.125752 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.125940 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:04.126021 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:04.126276 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 15:06:04.126405 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 15:06:04.126806 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:04.120671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:41901 (local address 127.24.103.65:43875)
0619 15:06:04.121278 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:04.121291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:04.121435 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:04.121778 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:04.121781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:04.121800 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:04.122100 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:04.122107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.123323 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.123328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:04.124812 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:04.124824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.125968 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.125974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:04.126121 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:04.126143 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 15:06:04.126240 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 15:06:04.126368 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":269,"threads_started":1}
I20260619 15:06:04.128008 6827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "384098af5dea42a58138c0eef02ee160" candidate_uuid: "9f573cf213df46329ff2000e66667ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1aea936947e41709523a572b8836648" is_pre_election: true
I20260619 15:06:04.128010 6693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "384098af5dea42a58138c0eef02ee160" candidate_uuid: "9f573cf213df46329ff2000e66667ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d416e496633a4b4ca187f7b0537adcca" is_pre_election: true
I20260619 15:06:04.128600 6693 raft_consensus.cc:2454] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f573cf213df46329ff2000e66667ad0 in term 0.
I20260619 15:06:04.128600 6827 raft_consensus.cc:2454] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f573cf213df46329ff2000e66667ad0 in term 0.
I20260619 15:06:04.129417 6494 leader_election.cc:302] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f573cf213df46329ff2000e66667ad0, f1aea936947e41709523a572b8836648; no voters:
I20260619 15:06:04.129787 6898 raft_consensus.cc:2781] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:06:04.129938 6898 raft_consensus.cc:482] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:06:04.130084 6898 raft_consensus.cc:3037] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:04.132447 6898 raft_consensus.cc:504] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } } peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } }
I20260619 15:06:04.133018 6898 leader_election.cc:288] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [CANDIDATE]: Term 1 election: Requested vote from peers d416e496633a4b4ca187f7b0537adcca (127.24.103.66:35171), f1aea936947e41709523a572b8836648 (127.24.103.67:41901)
I20260619 15:06:04.133596 6693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "384098af5dea42a58138c0eef02ee160" candidate_uuid: "9f573cf213df46329ff2000e66667ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d416e496633a4b4ca187f7b0537adcca"
I20260619 15:06:04.133625 6827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "384098af5dea42a58138c0eef02ee160" candidate_uuid: "9f573cf213df46329ff2000e66667ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1aea936947e41709523a572b8836648"
I20260619 15:06:04.133834 6693 raft_consensus.cc:3037] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:04.133829 6827 raft_consensus.cc:3037] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:04.137199 6827 raft_consensus.cc:2454] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f573cf213df46329ff2000e66667ad0 in term 1.
I20260619 15:06:04.137199 6693 raft_consensus.cc:2454] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f573cf213df46329ff2000e66667ad0 in term 1.
I20260619 15:06:04.137687 6494 leader_election.cc:302] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f573cf213df46329ff2000e66667ad0, f1aea936947e41709523a572b8836648; no voters:
I20260619 15:06:04.137989 6898 raft_consensus.cc:2781] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:06:04.138449 6898 raft_consensus.cc:686] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 1 LEADER]: Becoming Leader. State: Replica: 9f573cf213df46329ff2000e66667ad0, State: Running, Role: LEADER
I20260619 15:06:04.139000 6898 consensus_queue.cc:237] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } } peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } }
I20260619 15:06:04.143559 6414 catalog_manager.cc:5495] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f573cf213df46329ff2000e66667ad0 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "9f573cf213df46329ff2000e66667ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } health_report { overall_health: HEALTHY } } }
I20260619 15:06:04.161260 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:04.155588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.24.103.65:40079)
0619 15:06:04.155805 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:04.155810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:04.155830 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:06:04.156040 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:04.156043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:04.156178 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:04.156352 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:06:04.156732 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.157475 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.159085 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.159498 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.159640 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:04.160141 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:06:04.160655 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:06:04.160821 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:04.160877 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 15:06:04.160980 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 15:06:04.169595 6827 raft_consensus.cc:1261] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648 [term 1 FOLLOWER]: Refusing update from remote peer 9f573cf213df46329ff2000e66667ad0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:06:04.169700 6693 raft_consensus.cc:1261] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca [term 1 FOLLOWER]: Refusing update from remote peer 9f573cf213df46329ff2000e66667ad0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:06:04.170687 6901 consensus_queue.cc:1035] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:06:04.171134 6898 consensus_queue.cc:1035] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:06:04.200733 6415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:06:04.201025 6415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:06:04.204910 6415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Sending DeleteTablet for 3 replicas of tablet 384098af5dea42a58138c0eef02ee160
I20260619 15:06:04.206223 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 384098af5dea42a58138c0eef02ee160 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:04 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:47808
I20260619 15:06:04.206483 6807 tablet_service.cc:1514] Processing DeleteTablet for tablet 384098af5dea42a58138c0eef02ee160 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:04 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:45758
I20260619 15:06:04.206496 6673 tablet_service.cc:1514] Processing DeleteTablet for tablet 384098af5dea42a58138c0eef02ee160 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:04 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:50712
I20260619 15:06:04.206697 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 6476
I20260619 15:06:04.207695 6914 tablet_replica.cc:330] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0: stopping tablet replica
I20260619 15:06:04.208190 6914 raft_consensus.cc:2229] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:06:04.209180 6914 raft_consensus.cc:2258] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:06:04.211766 6914 ts_tablet_manager.cc:1905] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:06:04.218768 6915 tablet_replica.cc:330] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648: stopping tablet replica
I20260619 15:06:04.220038 6915 raft_consensus.cc:2229] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:06:04.221225 6915 raft_consensus.cc:2258] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:06:04.221928 6914 ts_tablet_manager.cc:1918] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:06:04.222416 6914 log.cc:1196] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/384098af5dea42a58138c0eef02ee160
I20260619 15:06:04.223122 6914 ts_tablet_manager.cc:1939] T 384098af5dea42a58138c0eef02ee160 P 9f573cf213df46329ff2000e66667ad0: Deleting consensus metadata
I20260619 15:06:04.224370 6915 ts_tablet_manager.cc:1905] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:06:04.225132 6402 catalog_manager.cc:4841] TS 9f573cf213df46329ff2000e66667ad0 (127.24.103.65:40079): tablet 384098af5dea42a58138c0eef02ee160 (table test-table [id=adf2a181dfac48a4a2b197d54e55831a]) successfully deleted
I20260619 15:06:04.226063 6916 tablet_replica.cc:330] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca: stopping tablet replica
I20260619 15:06:04.226503 6916 raft_consensus.cc:2229] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:06:04.227066 6916 raft_consensus.cc:2258] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 15:06:04 dist-test-slave-zpbw krb5kdc[6370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:06:04.234587 6915 ts_tablet_manager.cc:1918] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:06:04.234767 6915 log.cc:1196] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/384098af5dea42a58138c0eef02ee160
I20260619 15:06:04.235365 6915 ts_tablet_manager.cc:1939] T 384098af5dea42a58138c0eef02ee160 P f1aea936947e41709523a572b8836648: Deleting consensus metadata
I20260619 15:06:04.237355 6401 catalog_manager.cc:4841] TS f1aea936947e41709523a572b8836648 (127.24.103.67:41901): tablet 384098af5dea42a58138c0eef02ee160 (table test-table [id=adf2a181dfac48a4a2b197d54e55831a]) successfully deleted
I20260619 15:06:04.237443 6916 ts_tablet_manager.cc:1905] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:06:04.237711 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:04.207958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.24.103.65:40079)
0619 15:06:04.208142 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:04.208149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:04.219038 (+ 10889us) server_negotiation.cc:407] Connection header received
0619 15:06:04.219572 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:04.219579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:04.219681 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:04.219762 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:04.221819 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.223134 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.225164 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.225809 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.225888 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:04.230727 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:04.230739 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:04.230752 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:04.230862 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:04.234023 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:04.234888 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:04.234900 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:04.234904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:04.235104 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:04.235573 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:04.235578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:04.235581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:04.236064 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:04.236387 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:04.237019 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 15:06:04.237282 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 15:06:04.239436 6599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:04.245031 6916 ts_tablet_manager.cc:1918] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:06:04.245229 6916 log.cc:1196] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/384098af5dea42a58138c0eef02ee160
I20260619 15:06:04.245760 6916 ts_tablet_manager.cc:1939] T 384098af5dea42a58138c0eef02ee160 P d416e496633a4b4ca187f7b0537adcca: Deleting consensus metadata
I20260619 15:06:04.247756 6401 catalog_manager.cc:4841] TS d416e496633a4b4ca187f7b0537adcca (127.24.103.66:35171): tablet 384098af5dea42a58138c0eef02ee160 (table test-table [id=adf2a181dfac48a4a2b197d54e55831a]) successfully deleted
I20260619 15:06:04.340976 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 6476
I20260619 15:06:04.356396 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 6610
Jun 19 15:06:04 dist-test-slave-zpbw krb5kdc[6370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:06:04.372346 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:04.357348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.24.103.66:35171)
0619 15:06:04.357548 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:04.357556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:04.357594 (+ 38us) server_negotiation.cc:407] Connection header received
0619 15:06:04.357800 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:04.357806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:04.357880 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:04.357986 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:04.359459 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.360223 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.361481 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.361881 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.362022 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:04.366563 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:04.366579 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:04.366595 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:04.366641 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:04.369638 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:04.370357 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:04.370373 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:04.370376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:04.370530 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:04.370917 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:04.370939 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:04.370941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:04.371333 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:04.371594 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:04.371814 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 15:06:04.371984 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:06:04.373032 6733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:04.467967 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 6610
I20260619 15:06:04.484972 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 6744
Jun 19 15:06:04 dist-test-slave-zpbw krb5kdc[6370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:06:04.499892 6889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:04.485735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.24.103.67:41901)
0619 15:06:04.485929 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:04.485934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:04.485953 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:06:04.486065 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:04.486069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:04.486133 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:04.486225 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:04.487572 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.488273 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.489500 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:04.489894 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:04.489990 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:04.494274 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:04.494290 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:04.494306 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:04.494342 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:04.496994 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:04.497656 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:04.497663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:04.497665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:04.497818 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:04.498279 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:04.498286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:04.498289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:04.498656 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:04.498997 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:04.499325 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 15:06:04.499539 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:06:04.500514 6867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:04.590327 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 6744
I20260619 15:06:04.607048 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 6383
I20260619 15:06:04.608258 6445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:04.720192 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 6383
I20260619 15:06:04.741882 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44191
--webserver_interface=127.24.103.126
--webserver_port=34617
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:44191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:05.080063 6922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:05.080447 6922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:05.080617 6922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:05.088183 6922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:06:05.088351 6922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:05.088469 6922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:05.088567 6922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:06:05.088655 6922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:06:05.099058 6922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:44191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=34617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:05.100490 6922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:05.101894 6922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:05.125967 6928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:05.125957 6931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:05.125957 6929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:05.127034 6922 server_base.cc:1034] running on GCE node
I20260619 15:06:05.127986 6922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:05.129771 6922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:05.131031 6922 hybrid_clock.cc:648] HybridClock initialized: now 1781881565130955 us; error 61 us; skew 500 ppm
I20260619 15:06:05.131470 6922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:05 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881565, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:05.135331 6922 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:06:05.137327 6922 webserver.cc:460] Webserver started at http://127.24.103.126:34617/ using document root <none> and password file <none>
I20260619 15:06:05.138165 6922 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:05.138340 6922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:05.144788 6922 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 15:06:05.148054 6938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:05.149482 6922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 15:06:05.149703 6922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "e618e0c94f5c40e7a1b783f661673c08"
format_stamp: "Formatted at 2026-06-19 15:06:02 on dist-test-slave-zpbw"
server_key: "a4a9f86fc96168efb8e4620a2161ff36"
server_key_iv: "9e26b9d10b2cf8dc4ee83ebda0634eba"
server_key_version: "encryptionkey@0"
I20260619 15:06:05.150889 6922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:05.171381 6922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:05.176043 6922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:05.176362 6922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:05.196331 6922 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:44191
I20260619 15:06:05.196336 6989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:44191 every 8 connection(s)
I20260619 15:06:05.197948 6922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:06:05.205228 6990 sys_catalog.cc:263] Verifying existing consensus state
I20260619 15:06:05.206673 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 6922
I20260619 15:06:05.207810 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:40079
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:44191
--webserver_port=34843
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:06:05.210383 6990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Bootstrap starting.
I20260619 15:06:05.232657 6990 log.cc:824] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:05.245450 6990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:06:05.246070 6990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Bootstrap complete.
I20260619 15:06:05.255743 6990 raft_consensus.cc:348] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } }
I20260619 15:06:05.256748 6990 raft_consensus.cc:729] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e618e0c94f5c40e7a1b783f661673c08, State: Initialized, Role: FOLLOWER
I20260619 15:06:05.257499 6990 consensus_queue.cc:260] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } }
I20260619 15:06:05.257745 6990 raft_consensus.cc:388] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:06:05.257869 6990 raft_consensus.cc:482] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:06:05.258064 6990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 1 FOLLOWER]: Advancing to term 2
I20260619 15:06:05.262173 6990 raft_consensus.cc:504] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } }
I20260619 15:06:05.262878 6990 leader_election.cc:302] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e618e0c94f5c40e7a1b783f661673c08; no voters:
I20260619 15:06:05.263556 6990 leader_election.cc:288] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 15:06:05.263700 6994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 15:06:05.264912 6994 raft_consensus.cc:686] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 2 LEADER]: Becoming Leader. State: Replica: e618e0c94f5c40e7a1b783f661673c08, State: Running, Role: LEADER
I20260619 15:06:05.265604 6994 consensus_queue.cc:237] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } }
I20260619 15:06:05.266075 6990 sys_catalog.cc:564] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:06:05.268481 6995 sys_catalog.cc:455] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e618e0c94f5c40e7a1b783f661673c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } } }
I20260619 15:06:05.268934 6995 sys_catalog.cc:458] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: This master's current role is: LEADER
I20260619 15:06:05.273556 7000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:06:05.273487 6996 sys_catalog.cc:455] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e618e0c94f5c40e7a1b783f661673c08. Latest consensus state: current_term: 2 leader_uuid: "e618e0c94f5c40e7a1b783f661673c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } } }
I20260619 15:06:05.273917 6996 sys_catalog.cc:458] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: This master's current role is: LEADER
I20260619 15:06:05.282331 7000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:06:05.284081 7000 catalog_manager.cc:1259] Loaded cluster ID: 5e415f024e384b7781b300a54b953d70
I20260619 15:06:05.284282 7000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:06:05.287719 7000 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 15:06:05.306020 7000 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 15:06:05.316195 7011 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 15:06:05.327273 7000 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:06:05.329298 7000 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Loaded TSK: 0
I20260619 15:06:05.330941 7000 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 15:06:05.581161 6992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:05.581481 6992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:05.581626 6992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:05.589479 6992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:05.589612 6992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:05.589792 6992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:06:05.600395 6992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:40079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=34843
--tserver_master_addrs=127.24.103.126:44191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:05.601819 6992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:05.603353 6992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:05.618749 7018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:05.618839 7017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:05.619390 7020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:05.619979 6992 server_base.cc:1034] running on GCE node
I20260619 15:06:05.620638 6992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:05.621645 6992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:05.622915 6992 hybrid_clock.cc:648] HybridClock initialized: now 1781881565622829 us; error 57 us; skew 500 ppm
I20260619 15:06:05.623265 6992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:05 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881565, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:05.626629 6992 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:06:05.628432 6992 webserver.cc:460] Webserver started at http://127.24.103.65:34843/ using document root <none> and password file <none>
I20260619 15:06:05.629171 6992 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:05.629334 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:05.635303 6992 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 15:06:05.638576 7027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:05.640595 6992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 15:06:05.640802 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "9f573cf213df46329ff2000e66667ad0"
format_stamp: "Formatted at 2026-06-19 15:06:02 on dist-test-slave-zpbw"
server_key: "39f2da34ee05705cb0146d6ca094e2ba"
server_key_iv: "c2ab20bd0e6a25eddde8726c7fd7a6d8"
server_key_version: "encryptionkey@0"
I20260619 15:06:05.642213 6992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:05.680574 6992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:05.684772 6992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:05.685024 6992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:05.685953 6992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:05.687378 6992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:05.687477 6992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:05.687567 6992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:05.687626 6992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:05.721706 6992 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:40079
I20260619 15:06:05.721772 7139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:40079 every 8 connection(s)
I20260619 15:06:05.723292 6992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:06:05.725128 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 6992
I20260619 15:06:05.725883 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:35171
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:44191
--webserver_port=35005
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 15:06:05 dist-test-slave-zpbw krb5kdc[6370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881565, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:05.754014 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:05.726623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:42591)
0619 15:06:05.727583 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:05.727636 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:05.730445 (+ 2809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:05.732242 (+ 1797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:05.732256 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:05.733114 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:06:05.733917 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:05.733938 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:05.736724 (+ 2786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:05.736728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:05.738086 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:05.738101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:05.739719 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:05.739724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:05.739944 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:05.740742 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:06:05.740765 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:06:05.743394 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:06:05.748950 (+ 5556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:05.748970 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:05.749044 (+ 74us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:05.749669 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:05.750380 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:05.750384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:05.750387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:05.750639 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:05.751847 (+ 1208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:06:05.751854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:06:05.752203 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 15:06:05.752485 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 15:06:05.752951 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":372,"threads_started":1}
I20260619 15:06:05.754505 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:05.726324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:42591 (local address 127.24.103.126:44191)
0619 15:06:05.727025 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:05.727047 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:05.727793 (+ 746us) server_negotiation.cc:407] Connection header received
0619 15:06:05.731257 (+ 3464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:05.731275 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:05.731680 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:05.732009 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:05.734200 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:05.736474 (+ 2274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:05.738252 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:05.739456 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:05.739678 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:05.743780 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:05.743795 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:05.743821 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:05.743881 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:05.748657 (+ 4776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:05.749862 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:05.749876 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:05.749887 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:05.750184 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:05.750773 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:05.750810 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:05.750816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:05.751656 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:05.751888 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:05.752761 (+ 873us) server_negotiation.cc:299] Negotiation successful
0619 15:06:05.753321 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":277,"threads_started":1}
I20260619 15:06:05.757680 7140 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44191
I20260619 15:06:05.758040 7140 heartbeater.cc:464] Registering TS with master...
I20260619 15:06:05.758982 7140 heartbeater.cc:511] Master 127.24.103.126:44191 requested a full tablet report, sending...
I20260619 15:06:05.761592 6955 ts_manager.cc:194] Registered new tserver with Master: 9f573cf213df46329ff2000e66667ad0 (127.24.103.65:40079)
I20260619 15:06:05.763278 6955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:42591
W20260619 15:06:06.072052 7144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:06.072352 7144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:06.072510 7144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:06.080150 7144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:06.080281 7144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:06.080478 7144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:06:06.090600 7144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:35171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=35005
--tserver_master_addrs=127.24.103.126:44191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:06.091938 7144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:06.093279 7144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:06.106755 7149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:06.106735 7152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:06.106630 7150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:06.107508 7144 server_base.cc:1034] running on GCE node
I20260619 15:06:06.108134 7144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:06.109072 7144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:06.110298 7144 hybrid_clock.cc:648] HybridClock initialized: now 1781881566110249 us; error 46 us; skew 500 ppm
I20260619 15:06:06.110628 7144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:06 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881566, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:06.113677 7144 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:06:06.115202 7144 webserver.cc:460] Webserver started at http://127.24.103.66:35005/ using document root <none> and password file <none>
I20260619 15:06:06.115902 7144 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:06.116065 7144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:06.121644 7144 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260619 15:06:06.124460 7159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:06.125970 7144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 15:06:06.126154 7144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "d416e496633a4b4ca187f7b0537adcca"
format_stamp: "Formatted at 2026-06-19 15:06:03 on dist-test-slave-zpbw"
server_key: "d227d62a65da6cfa1175ea6a2df88d36"
server_key_iv: "8860e2e2ce256bf49bb5a6ef75de022a"
server_key_version: "encryptionkey@0"
I20260619 15:06:06.127265 7144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:06.155664 7144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:06.159844 7144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:06.160166 7144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:06.161165 7144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:06.162562 7144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:06.162689 7144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:06.162865 7144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:06.162959 7144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:06.199653 7144 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:35171
I20260619 15:06:06.199755 7271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:35171 every 8 connection(s)
I20260619 15:06:06.201213 7144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:06:06.203970 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 7144
I20260619 15:06:06.204766 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:41901
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:44191
--webserver_port=43317
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 15:06:06 dist-test-slave-zpbw krb5kdc[6370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881566, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:06.226390 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.204143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:60861 (local address 127.24.103.126:44191)
0619 15:06:06.204346 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:06.204352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:06.205661 (+ 1309us) server_negotiation.cc:407] Connection header received
0619 15:06:06.208004 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:06.208051 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:06.208188 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:06.208327 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:06.210016 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.210732 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.212490 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.213286 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.213413 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.218405 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:06.218415 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:06.218423 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:06.218471 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:06.222446 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:06.223485 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:06.223493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:06.223500 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:06.223666 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:06.224127 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:06.224136 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:06.224142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:06.224571 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:06.224747 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:06.225585 (+ 838us) server_negotiation.cc:299] Negotiation successful
0619 15:06:06.225920 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 15:06:06.227146 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.204686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:60861)
0619 15:06:06.205495 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:06.205545 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:06.207612 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:06.208626 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:06.208637 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:06.209224 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:06:06.209823 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.209843 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.210941 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.210946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.212281 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.212298 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.213802 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.213812 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.214155 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.214938 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:06:06.214969 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:06:06.218126 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:06:06.222707 (+ 4581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:06.222715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:06.222730 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:06.223338 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:06.223785 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:06.223789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:06.223791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:06.223981 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:06.224728 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:06:06.224735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:06:06.225170 (+ 435us) client_negotiation.cc:769] Sending connection context
0619 15:06:06.225411 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 15:06:06.225832 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":492,"threads_started":1}
I20260619 15:06:06.228772 7272 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44191
I20260619 15:06:06.229153 7272 heartbeater.cc:464] Registering TS with master...
I20260619 15:06:06.229907 7272 heartbeater.cc:511] Master 127.24.103.126:44191 requested a full tablet report, sending...
I20260619 15:06:06.231436 6955 ts_manager.cc:194] Registered new tserver with Master: d416e496633a4b4ca187f7b0537adcca (127.24.103.66:35171)
I20260619 15:06:06.232326 6955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:60861
W20260619 15:06:06.550101 7275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:06.550451 7275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:06.550618 7275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:06.558161 7275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:06.558342 7275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:06.558529 7275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:06:06.568809 7275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:41901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=43317
--tserver_master_addrs=127.24.103.126:44191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:06.570333 7275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:06.571676 7275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:06.585234 7280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:06.585198 7281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:06.585807 7283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:06.586299 7275 server_base.cc:1034] running on GCE node
I20260619 15:06:06.587093 7275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:06.588081 7275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:06.589308 7275 hybrid_clock.cc:648] HybridClock initialized: now 1781881566589243 us; error 60 us; skew 500 ppm
I20260619 15:06:06.589643 7275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:06 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881566, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:06.592798 7275 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:06:06.594223 7275 webserver.cc:460] Webserver started at http://127.24.103.67:43317/ using document root <none> and password file <none>
I20260619 15:06:06.595008 7275 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:06.595145 7275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:06.600430 7275 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:06:06.603183 7290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:06.604539 7275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:06:06.604717 7275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "f1aea936947e41709523a572b8836648"
format_stamp: "Formatted at 2026-06-19 15:06:03 on dist-test-slave-zpbw"
server_key: "14b75e4533e26ac3450a5cff77dc6dbb"
server_key_iv: "20d5b5ff8b38b3f1b32e4e145174970f"
server_key_version: "encryptionkey@0"
I20260619 15:06:06.605692 7275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:06.637180 7275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:06.641283 7275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:06.641584 7275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:06.642607 7275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:06.644027 7275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:06.644126 7275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:06.644291 7275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:06.644392 7275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:06.679054 7275 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:41901
I20260619 15:06:06.679160 7402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:41901 every 8 connection(s)
I20260619 15:06:06.680665 7275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:06:06.684145 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 7275
Jun 19 15:06:06 dist-test-slave-zpbw krb5kdc[6370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881566, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:06.705000 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.685197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.24.103.126:44191)
0619 15:06:06.687174 (+ 1977us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:06.687179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:06.689395 (+ 2216us) server_negotiation.cc:407] Connection header received
0619 15:06:06.689519 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:06.689526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:06.689590 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:06.689699 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:06.691234 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.692263 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.694161 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.694992 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.695210 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.698682 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:06.698689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:06.698693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:06.698734 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:06.701810 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:06.702654 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:06.702658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:06.702663 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:06.702878 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:06.703348 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:06.703355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:06.703361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:06.703728 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:06.703991 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:06.704365 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 15:06:06.704609 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":1882,"spinlock_wait_cycles":13312,"thread_start_us":1763,"threads_started":1}
I20260619 15:06:06.706897 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.683413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:49399 (local address 127.24.103.126:44191)
0619 15:06:06.683597 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:06.683602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:06.685180 (+ 1578us) server_negotiation.cc:407] Connection header received
0619 15:06:06.688772 (+ 3592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:06.688777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:06.688851 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:06.688990 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:06.690811 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.691906 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.694227 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.695008 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.695210 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.699481 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:06.699489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:06.699494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:06.699544 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:06.703121 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:06.704065 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:06.704072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:06.704076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:06.704220 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:06.704890 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:06.704894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:06.704896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:06.705239 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:06.705369 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:06.706308 (+ 939us) server_negotiation.cc:299] Negotiation successful
0619 15:06:06.706545 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":62,"spinlock_wait_cycles":896}
I20260619 15:06:06.708220 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.684117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:49399)
0619 15:06:06.685015 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:06.685068 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:06.688344 (+ 3276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:06.689257 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:06.689265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:06.690028 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:06:06.690620 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.690638 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.692093 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.692097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.694092 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.694109 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.695260 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.695267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.695589 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.696238 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:06:06.696269 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:06:06.699228 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:06:06.703255 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:06.703263 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:06.703277 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:06.703945 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:06.704354 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:06.704358 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:06.704361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:06.704618 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:06.705391 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:06:06.705399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:06:06.705890 (+ 491us) client_negotiation.cc:769] Sending connection context
0619 15:06:06.706249 (+ 359us) client_negotiation.cc:240] Negotiation successful
0619 15:06:06.706853 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":338,"threads_started":1}
I20260619 15:06:06.709584 7403 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44191
I20260619 15:06:06.709877 7403 heartbeater.cc:464] Registering TS with master...
I20260619 15:06:06.710676 7403 heartbeater.cc:511] Master 127.24.103.126:44191 requested a full tablet report, sending...
I20260619 15:06:06.712109 6955 ts_manager.cc:194] Registered new tserver with Master: f1aea936947e41709523a572b8836648 (127.24.103.67:41901)
I20260619 15:06:06.712891 6955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:49399
I20260619 15:06:06.720465 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:06:06.728531 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.721594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.24.103.126:44191)
0619 15:06:06.721798 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:06.721805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:06.722176 (+ 371us) server_negotiation.cc:407] Connection header received
0619 15:06:06.722381 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:06.722388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:06.722599 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:06.722741 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:06:06.723228 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.724366 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.726075 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.726733 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.726936 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.727416 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:06:06.728039 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:06:06.728148 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:06.728254 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 15:06:06.728328 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 15:06:06.729534 6955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:06:06.736965 6955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:06:06.761808 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.751150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.24.103.66:35171)
0619 15:06:06.752130 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:06.752144 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:06.752876 (+ 732us) server_negotiation.cc:407] Connection header received
0619 15:06:06.753003 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:06.753010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:06.753299 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:06.753512 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:06.754757 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.756409 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.759399 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.760557 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.760728 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.760946 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:06.761278 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 15:06:06.761455 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":1102,"threads_started":1}
I20260619 15:06:06.761842 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.749557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:35171 (local address 127.0.0.1:42674)
0619 15:06:06.752600 (+ 3043us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:06.752621 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:06.752748 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:06.753807 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:06.753815 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:06.753852 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:06.754513 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.754530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.756557 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.756562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.759200 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.759215 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.760802 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.760809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.761001 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.761068 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 15:06:06.761214 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 15:06:06.761419 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2909,"mutex_wait_us":67,"thread_start_us":2770,"threads_started":1}
I20260619 15:06:06.762984 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.749649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.24.103.65:40079)
0619 15:06:06.753002 (+ 3353us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:06.753014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:06.753037 (+ 23us) server_negotiation.cc:407] Connection header received
0619 15:06:06.753112 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:06.753117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:06.753296 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:06.753948 (+ 652us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:06.754503 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.756139 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.759922 (+ 3783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.761449 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.761591 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.761816 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:06.762409 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 15:06:06.762582 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3208,"thread_start_us":3042,"threads_started":1}
I20260619 15:06:06.763757 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.749460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:40079 (local address 127.0.0.1:53410)
0619 15:06:06.750367 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:06.750441 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:06.750667 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:06.753581 (+ 2914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:06.753589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:06.753631 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:06.754237 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.754258 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.756263 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.756271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.759736 (+ 3465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.759749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.761992 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.762001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.762172 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.762201 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 15:06:06.763196 (+ 995us) client_negotiation.cc:240] Negotiation successful
0619 15:06:06.763337 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":221,"thread_start_us":395,"threads_started":1}
I20260619 15:06:06.763947 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.753492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:41901 (local address 127.0.0.1:46842)
0619 15:06:06.755851 (+ 2359us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:06.755870 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:06.755985 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:06.756630 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:06.756634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:06.756662 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:06.757174 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.757185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.758863 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.758881 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.760948 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.760964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.762974 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.762980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.763265 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.763301 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 15:06:06.763384 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 15:06:06.763542 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2228,"mutex_wait_us":25,"thread_start_us":2040,"threads_started":1}
I20260619 15:06:06.764685 7417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.753087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.24.103.67:41901)
0619 15:06:06.753841 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:06.753849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:06.755988 (+ 2139us) server_negotiation.cc:407] Connection header received
0619 15:06:06.756150 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:06.756158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:06.756443 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:06.756638 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:06.757364 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.758687 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.761127 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.762578 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.762695 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.764117 (+ 1422us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:06.764198 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 15:06:06.764363 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":528,"threads_started":1}
I20260619 15:06:06.766292 7207 tablet_service.cc:1467] Processing CreateTablet for tablet 14ed13f0125b4cf1aaebbc8b08e761bf (DEFAULT_TABLE table=test-table [id=3e1e2117e8cd4c5188fed2fd3856b64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:06:06.767352 7140 heartbeater.cc:503] Master 127.24.103.126:44191 was elected leader, sending a full tablet report...
I20260619 15:06:06.767531 7075 tablet_service.cc:1467] Processing CreateTablet for tablet 14ed13f0125b4cf1aaebbc8b08e761bf (DEFAULT_TABLE table=test-table [id=3e1e2117e8cd4c5188fed2fd3856b64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:06:06.768227 7338 tablet_service.cc:1467] Processing CreateTablet for tablet 14ed13f0125b4cf1aaebbc8b08e761bf (DEFAULT_TABLE table=test-table [id=3e1e2117e8cd4c5188fed2fd3856b64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:06:06.769079 7207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14ed13f0125b4cf1aaebbc8b08e761bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:06.769595 7338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14ed13f0125b4cf1aaebbc8b08e761bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:06.769896 7075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14ed13f0125b4cf1aaebbc8b08e761bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:06.784068 7419 tablet_bootstrap.cc:492] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648: Bootstrap starting.
I20260619 15:06:06.786433 7420 tablet_bootstrap.cc:492] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0: Bootstrap starting.
I20260619 15:06:06.787792 7419 tablet_bootstrap.cc:654] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:06.789132 7419 log.cc:824] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:06.790156 7420 tablet_bootstrap.cc:654] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:06.790630 7421 tablet_bootstrap.cc:492] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca: Bootstrap starting.
I20260619 15:06:06.791471 7420 log.cc:824] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:06.792904 7419 tablet_bootstrap.cc:492] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648: No bootstrap required, opened a new log
I20260619 15:06:06.793234 7419 ts_tablet_manager.cc:1397] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 15:06:06.794821 7420 tablet_bootstrap.cc:492] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0: No bootstrap required, opened a new log
I20260619 15:06:06.795090 7420 ts_tablet_manager.cc:1397] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 15:06:06.795367 7421 tablet_bootstrap.cc:654] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:06.796722 7421 log.cc:824] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:06.800235 7421 tablet_bootstrap.cc:492] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca: No bootstrap required, opened a new log
I20260619 15:06:06.800539 7421 ts_tablet_manager.cc:1397] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 15:06:06.803649 7420 raft_consensus.cc:348] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } } peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } }
I20260619 15:06:06.804064 7420 raft_consensus.cc:374] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:06.804169 7420 raft_consensus.cc:729] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f573cf213df46329ff2000e66667ad0, State: Initialized, Role: FOLLOWER
I20260619 15:06:06.804833 7420 consensus_queue.cc:260] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } } peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } }
I20260619 15:06:06.805121 7419 raft_consensus.cc:348] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } } peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } }
I20260619 15:06:06.805625 7419 raft_consensus.cc:374] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:06.805737 7419 raft_consensus.cc:729] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1aea936947e41709523a572b8836648, State: Initialized, Role: FOLLOWER
I20260619 15:06:06.806485 7419 consensus_queue.cc:260] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } } peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } }
I20260619 15:06:06.806994 7420 ts_tablet_manager.cc:1428] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 15:06:06.807984 7403 heartbeater.cc:503] Master 127.24.103.126:44191 was elected leader, sending a full tablet report...
I20260619 15:06:06.808617 7419 ts_tablet_manager.cc:1428] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260619 15:06:06.812217 7421 raft_consensus.cc:348] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } } peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } }
I20260619 15:06:06.812779 7421 raft_consensus.cc:374] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:06.812920 7421 raft_consensus.cc:729] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d416e496633a4b4ca187f7b0537adcca, State: Initialized, Role: FOLLOWER
I20260619 15:06:06.813557 7421 consensus_queue.cc:260] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } } peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } }
I20260619 15:06:06.814582 7272 heartbeater.cc:503] Master 127.24.103.126:44191 was elected leader, sending a full tablet report...
I20260619 15:06:06.815201 7421 ts_tablet_manager.cc:1428] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
W20260619 15:06:06.932364 7404 tablet.cc:2369] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:06:06.957407 7273 tablet.cc:2369] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:06:06.975807 7141 tablet.cc:2369] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:06:06.983137 7427 raft_consensus.cc:482] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:06:06.983414 7427 raft_consensus.cc:504] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } } peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } }
I20260619 15:06:06.985001 7427 leader_election.cc:288] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f573cf213df46329ff2000e66667ad0 (127.24.103.65:40079), f1aea936947e41709523a572b8836648 (127.24.103.67:41901)
I20260619 15:06:06.990893 7426 raft_consensus.cc:482] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:06:06.991237 7426 raft_consensus.cc:504] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } } peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } }
I20260619 15:06:06.992616 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.985472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:39219 (local address 127.24.103.65:40079)
0619 15:06:06.985730 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:06.985748 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:06.986316 (+ 568us) server_negotiation.cc:407] Connection header received
0619 15:06:06.986624 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:06.986628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:06.986689 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:06.986787 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:06.987400 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.988115 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.990256 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.991541 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.991708 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.991792 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:06.992203 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 15:06:06.992356 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 15:06:06.992784 7417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.985580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:39431 (local address 127.24.103.67:41901)
0619 15:06:06.985952 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:06.985961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:06.986665 (+ 704us) server_negotiation.cc:407] Connection header received
0619 15:06:06.986897 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:06.986901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:06.986959 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:06.987058 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:06.987618 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.988337 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.990309 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.991659 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.991802 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.991880 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:06.992384 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 15:06:06.992526 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260619 15:06:06.993228 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.985257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:40079 (local address 127.24.103.66:39219)
0619 15:06:06.986191 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:06.986207 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:06.986456 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:06.986873 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:06.986878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:06.986903 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:06.987248 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.987256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.988257 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.988265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.990107 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.990119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.991758 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.991765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.991966 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.992030 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 15:06:06.992142 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 15:06:06.992289 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":33,"thread_start_us":932,"threads_started":1}
I20260619 15:06:06.993419 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.985322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:41901 (local address 127.24.103.66:39431)
0619 15:06:06.986541 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:06.986555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:06.986719 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:06.987078 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:06.987082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:06.987107 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:06.987474 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.987483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.988481 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.988492 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.990120 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.990134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.991963 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.991968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.992201 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.992236 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 15:06:06.992961 (+ 725us) client_negotiation.cc:240] Negotiation successful
0619 15:06:06.993068 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1110,"mutex_wait_us":33,"thread_start_us":506,"threads_started":1}
I20260619 15:06:06.995221 7358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ed13f0125b4cf1aaebbc8b08e761bf" candidate_uuid: "d416e496633a4b4ca187f7b0537adcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1aea936947e41709523a572b8836648" is_pre_election: true
I20260619 15:06:06.995292 7095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ed13f0125b4cf1aaebbc8b08e761bf" candidate_uuid: "d416e496633a4b4ca187f7b0537adcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f573cf213df46329ff2000e66667ad0" is_pre_election: true
I20260619 15:06:06.995762 7358 raft_consensus.cc:2454] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d416e496633a4b4ca187f7b0537adcca in term 0.
I20260619 15:06:06.995903 7095 raft_consensus.cc:2454] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d416e496633a4b4ca187f7b0537adcca in term 0.
I20260619 15:06:06.996562 7162 leader_election.cc:302] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d416e496633a4b4ca187f7b0537adcca, f1aea936947e41709523a572b8836648; no voters:
I20260619 15:06:06.997289 7427 raft_consensus.cc:2781] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:06:06.997435 7427 raft_consensus.cc:482] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:06:06.997545 7427 raft_consensus.cc:3037] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:06.998898 7426 leader_election.cc:288] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f573cf213df46329ff2000e66667ad0 (127.24.103.65:40079), d416e496633a4b4ca187f7b0537adcca (127.24.103.66:35171)
I20260619 15:06:07.000684 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.994000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:36385 (local address 127.24.103.65:40079)
0619 15:06:06.994181 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:06.994186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:06.994202 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:06:06.994259 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:06.994267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:06.994346 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:06.994433 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:06.995730 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.996921 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.998713 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.999682 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.999875 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:06.999950 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:07.000252 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 15:06:07.000392 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 15:06:07.000715 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.993532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:40079 (local address 127.24.103.67:36385)
0619 15:06:06.993920 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:06.993940 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:06.994047 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:06.995230 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:06.995234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:06.995265 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:06.995549 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.995566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.997053 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.997059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:06.998549 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.998561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.999847 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.999853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:07.000070 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:07.000101 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 15:06:07.000227 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 15:06:07.000376 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":26,"spinlock_wait_cycles":21888}
I20260619 15:06:07.000880 7427 raft_consensus.cc:504] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } } peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } }
I20260619 15:06:07.001549 7427 leader_election.cc:288] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [CANDIDATE]: Term 1 election: Requested vote from peers 9f573cf213df46329ff2000e66667ad0 (127.24.103.65:40079), f1aea936947e41709523a572b8836648 (127.24.103.67:41901)
I20260619 15:06:07.001986 7095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ed13f0125b4cf1aaebbc8b08e761bf" candidate_uuid: "d416e496633a4b4ca187f7b0537adcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f573cf213df46329ff2000e66667ad0"
I20260619 15:06:07.002224 7095 raft_consensus.cc:3037] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:07.002367 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.994010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:50483 (local address 127.24.103.66:35171)
0619 15:06:06.994225 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:06.994233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:06.995251 (+ 1018us) server_negotiation.cc:407] Connection header received
0619 15:06:06.995526 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:06.995531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:06.995605 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:06.995671 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:06.996636 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.997693 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:07.000211 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:07.001353 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:07.001544 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:07.001619 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:07.001997 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 15:06:07.002127 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:06:07.002523 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:06.993641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:35171 (local address 127.24.103.67:50483)
0619 15:06:06.995149 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:06.995169 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:06.995368 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:06.995858 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:06.995863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:06.995896 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:06.996327 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:06.996340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:06.997897 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:06.997905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:07.000037 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:07.000051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:07.001593 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:07.001599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:07.001786 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:07.001827 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 15:06:07.001962 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 15:06:07.002107 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1337,"mutex_wait_us":1,"thread_start_us":1065,"threads_started":1}
I20260619 15:06:07.002321 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ed13f0125b4cf1aaebbc8b08e761bf" candidate_uuid: "f1aea936947e41709523a572b8836648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f573cf213df46329ff2000e66667ad0" is_pre_election: true
I20260619 15:06:07.003720 7358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ed13f0125b4cf1aaebbc8b08e761bf" candidate_uuid: "d416e496633a4b4ca187f7b0537adcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1aea936947e41709523a572b8836648"
I20260619 15:06:07.003949 7358 raft_consensus.cc:3037] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:07.003806 7227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ed13f0125b4cf1aaebbc8b08e761bf" candidate_uuid: "f1aea936947e41709523a572b8836648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d416e496633a4b4ca187f7b0537adcca" is_pre_election: true
I20260619 15:06:07.004323 7227 raft_consensus.cc:2379] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1aea936947e41709523a572b8836648 in current term 1: Already voted for candidate d416e496633a4b4ca187f7b0537adcca in this term.
I20260619 15:06:07.005205 7293 leader_election.cc:302] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1aea936947e41709523a572b8836648; no voters: 9f573cf213df46329ff2000e66667ad0, d416e496633a4b4ca187f7b0537adcca
I20260619 15:06:07.005442 7095 raft_consensus.cc:2454] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d416e496633a4b4ca187f7b0537adcca in term 1.
I20260619 15:06:07.005860 7163 leader_election.cc:302] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f573cf213df46329ff2000e66667ad0, d416e496633a4b4ca187f7b0537adcca; no voters:
I20260619 15:06:07.006151 7427 raft_consensus.cc:2781] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:06:07.006641 7427 raft_consensus.cc:686] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 1 LEADER]: Becoming Leader. State: Replica: d416e496633a4b4ca187f7b0537adcca, State: Running, Role: LEADER
I20260619 15:06:07.007254 7358 raft_consensus.cc:2454] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d416e496633a4b4ca187f7b0537adcca in term 1.
I20260619 15:06:07.007193 7427 consensus_queue.cc:237] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } } peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } }
I20260619 15:06:07.007407 7426 raft_consensus.cc:2726] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 15:06:07.011921 6955 catalog_manager.cc:5495] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca reported cstate change: term changed from 0 to 1, leader changed from <none> to d416e496633a4b4ca187f7b0537adcca (127.24.103.66). New cstate: current_term: 1 leader_uuid: "d416e496633a4b4ca187f7b0537adcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d416e496633a4b4ca187f7b0537adcca" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 } health_report { overall_health: UNKNOWN } } }
I20260619 15:06:07.064930 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:07.059633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.24.103.66:35171)
0619 15:06:07.059820 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:07.059824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:07.059866 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:06:07.059981 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:07.059985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:07.060080 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:07.060211 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:06:07.060653 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:07.061378 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:07.062769 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:07.063258 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:07.063356 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:07.063833 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:06:07.064345 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:06:07.064462 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:07.064532 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 15:06:07.064597 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 15:06:07.073531 7095 raft_consensus.cc:1261] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0 [term 1 FOLLOWER]: Refusing update from remote peer d416e496633a4b4ca187f7b0537adcca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:06:07.073581 7358 raft_consensus.cc:1261] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [term 1 FOLLOWER]: Refusing update from remote peer d416e496633a4b4ca187f7b0537adcca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:06:07.074491 7431 consensus_queue.cc:1035] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f573cf213df46329ff2000e66667ad0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:06:07.074900 7427 consensus_queue.cc:1035] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1aea936947e41709523a572b8836648" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:06:07.102285 6955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:06:07.102603 6955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:06:07.105985 6955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Sending DeleteTablet for 3 replicas of tablet 14ed13f0125b4cf1aaebbc8b08e761bf
I20260619 15:06:07.107275 7338 tablet_service.cc:1514] Processing DeleteTablet for tablet 14ed13f0125b4cf1aaebbc8b08e761bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:07 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:46842
I20260619 15:06:07.107290 7075 tablet_service.cc:1514] Processing DeleteTablet for tablet 14ed13f0125b4cf1aaebbc8b08e761bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:07 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:53410
I20260619 15:06:07.107489 7207 tablet_service.cc:1514] Processing DeleteTablet for tablet 14ed13f0125b4cf1aaebbc8b08e761bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:07 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:42674
I20260619 15:06:07.108633 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 6992
I20260619 15:06:07.118610 7446 tablet_replica.cc:330] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca: stopping tablet replica
I20260619 15:06:07.119192 7446 raft_consensus.cc:2229] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:06:07.120024 7446 raft_consensus.cc:2258] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:06:07.122875 7446 ts_tablet_manager.cc:1905] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:06:07.131371 7445 tablet_replica.cc:330] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0: stopping tablet replica
I20260619 15:06:07.131884 7445 raft_consensus.cc:2229] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:06:07.132506 7445 raft_consensus.cc:2258] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:06:07.134394 7446 ts_tablet_manager.cc:1918] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:06:07.134586 7446 log.cc:1196] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/14ed13f0125b4cf1aaebbc8b08e761bf
I20260619 15:06:07.134797 7445 ts_tablet_manager.cc:1905] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:06:07.135270 7446 ts_tablet_manager.cc:1939] T 14ed13f0125b4cf1aaebbc8b08e761bf P d416e496633a4b4ca187f7b0537adcca: Deleting consensus metadata
I20260619 15:06:07.137207 6941 catalog_manager.cc:4841] TS d416e496633a4b4ca187f7b0537adcca (127.24.103.66:35171): tablet 14ed13f0125b4cf1aaebbc8b08e761bf (table test-table [id=3e1e2117e8cd4c5188fed2fd3856b64b]) successfully deleted
I20260619 15:06:07.138096 7444 tablet_replica.cc:330] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648: stopping tablet replica
I20260619 15:06:07.138742 7444 raft_consensus.cc:2229] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:06:07.139304 7444 raft_consensus.cc:2258] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:06:07.141225 7445 ts_tablet_manager.cc:1918] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:06:07.141412 7445 log.cc:1196] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/14ed13f0125b4cf1aaebbc8b08e761bf
I20260619 15:06:07.141427 7444 ts_tablet_manager.cc:1905] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:06:07.141875 7445 ts_tablet_manager.cc:1939] T 14ed13f0125b4cf1aaebbc8b08e761bf P 9f573cf213df46329ff2000e66667ad0: Deleting consensus metadata
I20260619 15:06:07.146669 6942 catalog_manager.cc:4841] TS 9f573cf213df46329ff2000e66667ad0 (127.24.103.65:40079): tablet 14ed13f0125b4cf1aaebbc8b08e761bf (table test-table [id=3e1e2117e8cd4c5188fed2fd3856b64b]) successfully deleted
I20260619 15:06:07.148739 7444 ts_tablet_manager.cc:1918] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:06:07.148905 7444 log.cc:1196] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/14ed13f0125b4cf1aaebbc8b08e761bf
I20260619 15:06:07.149444 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:07.110469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.24.103.65:40079)
0619 15:06:07.110700 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:07.110708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:07.110732 (+ 24us) server_negotiation.cc:407] Connection header received
0619 15:06:07.110830 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:07.110837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:07.110968 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:07.111099 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:07.113744 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:07.115324 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:07.118466 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:07.119160 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:07.119304 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:07.143135 (+ 23831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:07.143149 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:07.143165 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:07.143221 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:07.146076 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:07.147120 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:07.147131 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:07.147135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:07.147284 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:07.147771 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:07.147778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:07.147781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:07.148129 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:07.148438 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:07.148830 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 15:06:07.149058 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 15:06:07.149462 7444 ts_tablet_manager.cc:1939] T 14ed13f0125b4cf1aaebbc8b08e761bf P f1aea936947e41709523a572b8836648: Deleting consensus metadata
I20260619 15:06:07.150717 7135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:07.151465 6941 catalog_manager.cc:4841] TS f1aea936947e41709523a572b8836648 (127.24.103.67:41901): tablet 14ed13f0125b4cf1aaebbc8b08e761bf (table test-table [id=3e1e2117e8cd4c5188fed2fd3856b64b]) successfully deleted
I20260619 15:06:07.273607 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 6992
I20260619 15:06:07.298125 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 7144
I20260619 15:06:07.313688 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:07.298975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.24.103.66:35171)
0619 15:06:07.299278 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:07.299284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:07.299305 (+ 21us) server_negotiation.cc:407] Connection header received
0619 15:06:07.299433 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:07.299439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:07.299521 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:07.299644 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:07.300959 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:07.301756 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:07.303328 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:07.303797 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:07.303948 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:07.307023 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:07.307039 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:07.307056 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:07.307100 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:07.310564 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:07.311478 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:07.311486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:07.311489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:07.311645 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:07.312050 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:07.312062 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:07.312064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:07.312476 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:07.312813 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:07.313110 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 15:06:07.313343 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260619 15:06:07.314565 7267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:07.430970 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 7144
I20260619 15:06:07.450894 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 7275
I20260619 15:06:07.464496 7417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:07.451740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.24.103.67:41901)
0619 15:06:07.451940 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:07.451944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:07.451962 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:06:07.452133 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:07.452137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:07.452199 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:07.452283 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:07.453603 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:07.454321 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:07.455634 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:07.456092 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:07.456185 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:07.458874 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:07.458890 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:07.458905 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:07.458960 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:07.461658 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:07.462560 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:07.462567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:07.462569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:07.462705 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:07.463109 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:07.463113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:07.463115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:07.463472 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:07.463690 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:07.463946 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 15:06:07.464126 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 15:06:07.465139 7398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:07.555990 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 7275
I20260619 15:06:07.573194 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 6922
I20260619 15:06:07.574283 6985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:07.688328 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 6922
I20260619 15:06:07.713219 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44191
--webserver_interface=127.24.103.126
--webserver_port=34617
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:44191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:08.048208 7451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:08.048607 7451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:08.048786 7451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:08.056226 7451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:06:08.056411 7451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:08.056527 7451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:08.056632 7451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:06:08.056720 7451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:06:08.066850 7451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:44191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=34617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:08.068270 7451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:08.069785 7451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:08.082345 7459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:08.082361 7456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:08.082392 7457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:08.083011 7451 server_base.cc:1034] running on GCE node
I20260619 15:06:08.083895 7451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:08.085403 7451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:08.086653 7451 hybrid_clock.cc:648] HybridClock initialized: now 1781881568086604 us; error 65 us; skew 500 ppm
I20260619 15:06:08.087056 7451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:08 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881568, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:08.090337 7451 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:06:08.091890 7451 webserver.cc:460] Webserver started at http://127.24.103.126:34617/ using document root <none> and password file <none>
I20260619 15:06:08.092561 7451 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:08.092660 7451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:08.098294 7451 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 15:06:08.101217 7466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:08.102623 7451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 15:06:08.102830 7451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "e618e0c94f5c40e7a1b783f661673c08"
format_stamp: "Formatted at 2026-06-19 15:06:02 on dist-test-slave-zpbw"
server_key: "a4a9f86fc96168efb8e4620a2161ff36"
server_key_iv: "9e26b9d10b2cf8dc4ee83ebda0634eba"
server_key_version: "encryptionkey@0"
I20260619 15:06:08.103852 7451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:08.126031 7451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:08.130724 7451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:08.130985 7451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:08.150142 7451 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:44191
I20260619 15:06:08.150154 7517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:44191 every 8 connection(s)
I20260619 15:06:08.151820 7451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:06:08.158490 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 7451
I20260619 15:06:08.159289 7518 sys_catalog.cc:263] Verifying existing consensus state
I20260619 15:06:08.159615 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:40079
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:44191
--webserver_port=34843
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:06:08.164757 7518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Bootstrap starting.
I20260619 15:06:08.186223 7518 log.cc:824] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:08.202108 7518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:06:08.202657 7518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08: Bootstrap complete.
I20260619 15:06:08.212342 7518 raft_consensus.cc:348] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } }
I20260619 15:06:08.213253 7518 raft_consensus.cc:729] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e618e0c94f5c40e7a1b783f661673c08, State: Initialized, Role: FOLLOWER
I20260619 15:06:08.213961 7518 consensus_queue.cc:260] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } }
I20260619 15:06:08.214195 7518 raft_consensus.cc:388] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:06:08.214327 7518 raft_consensus.cc:482] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:06:08.214514 7518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 2 FOLLOWER]: Advancing to term 3
I20260619 15:06:08.218421 7518 raft_consensus.cc:504] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } }
I20260619 15:06:08.218988 7518 leader_election.cc:302] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e618e0c94f5c40e7a1b783f661673c08; no voters:
I20260619 15:06:08.219707 7518 leader_election.cc:288] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 15:06:08.219848 7522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 15:06:08.221099 7522 raft_consensus.cc:686] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [term 3 LEADER]: Becoming Leader. State: Replica: e618e0c94f5c40e7a1b783f661673c08, State: Running, Role: LEADER
I20260619 15:06:08.221714 7522 consensus_queue.cc:237] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } }
I20260619 15:06:08.221925 7518 sys_catalog.cc:564] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:06:08.224303 7524 sys_catalog.cc:455] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e618e0c94f5c40e7a1b783f661673c08. Latest consensus state: current_term: 3 leader_uuid: "e618e0c94f5c40e7a1b783f661673c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } } }
I20260619 15:06:08.224725 7524 sys_catalog.cc:458] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: This master's current role is: LEADER
I20260619 15:06:08.225283 7523 sys_catalog.cc:455] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e618e0c94f5c40e7a1b783f661673c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618e0c94f5c40e7a1b783f661673c08" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44191 } } }
I20260619 15:06:08.225616 7523 sys_catalog.cc:458] T 00000000000000000000000000000000 P e618e0c94f5c40e7a1b783f661673c08 [sys.catalog]: This master's current role is: LEADER
I20260619 15:06:08.227775 7528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:06:08.240657 7528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:06:08.242846 7528 catalog_manager.cc:1259] Loaded cluster ID: 5e415f024e384b7781b300a54b953d70
I20260619 15:06:08.243081 7528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 15:06:08.245820 7528 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbf0dae09ad google::LogMessage::Fail() at ??:0
@ 0x7fbf0dae388a google::LogMessage::SendToLog() at ??:0
@ 0x7fbf0dae04de google::LogMessage::Flush() at ??:0
@ 0x7fbf0dae1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbf23513512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fbf23510976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbf23598610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fbf23598439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbf1072c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbf1072f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbf1072f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbf1070aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbf139e46db start_thread at ??:0
@ 0x7fbf0c97a61f clone at ??:0
W20260619 15:06:08.542718 7520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:08.543144 7520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:08.543310 7520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:08.550891 7520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:08.551060 7520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:08.551270 7520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:06:08.561451 7520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:40079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=34843
--tserver_master_addrs=127.24.103.126:44191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:08.562866 7520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:08.564354 7520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:08.577950 7558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:08.577975 7557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:08.577889 7560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:08.578734 7520 server_base.cc:1034] running on GCE node
I20260619 15:06:08.579377 7520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:08.580328 7520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:08.581564 7520 hybrid_clock.cc:648] HybridClock initialized: now 1781881568581510 us; error 47 us; skew 500 ppm
I20260619 15:06:08.581897 7520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:08 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881568, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:08.585528 7520 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:06:08.587102 7520 webserver.cc:460] Webserver started at http://127.24.103.65:34843/ using document root <none> and password file <none>
I20260619 15:06:08.587816 7520 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:08.587953 7520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:08.593254 7520 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:06:08.596016 7567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:08.597361 7520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:06:08.597541 7520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "9f573cf213df46329ff2000e66667ad0"
format_stamp: "Formatted at 2026-06-19 15:06:02 on dist-test-slave-zpbw"
server_key: "39f2da34ee05705cb0146d6ca094e2ba"
server_key_iv: "c2ab20bd0e6a25eddde8726c7fd7a6d8"
server_key_version: "encryptionkey@0"
I20260619 15:06:08.598649 7520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:08.624199 7520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:08.628821 7520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:08.629118 7520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:08.630194 7520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:08.631619 7520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:08.631717 7520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:08.631915 7520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:08.632006 7520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:08.666618 7520 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:40079
I20260619 15:06:08.666710 7679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:40079 every 8 connection(s)
I20260619 15:06:08.668177 7520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:06:08.673314 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:08.671446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:47811)
0619 15:06:08.672241 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:08.672470 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":308,"threads_started":1}
W20260619 15:06:08.673980 7680 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:44191 (0 consecutive failures): Network error: Failed to ping master at 127.24.103.126:44191: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
I20260619 15:06:08.674747 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:08.674434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:46359)
0619 15:06:08.674580 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:08.674648 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 15:06:08.675588 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:08.675284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:50873)
0619 15:06:08.675406 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:08.675477 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 15:06:08.675839 7680 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 15:06:08.676471 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 7520
I20260619 15:06:08.677188 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:35171
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:44191
--webserver_port=35005
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:09.013226 7683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:09.013533 7683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:09.013701 7683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:09.021275 7683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:09.021399 7683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:09.021585 7683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:06:09.031930 7683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:35171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=35005
--tserver_master_addrs=127.24.103.126:44191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:09.033334 7683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:09.034713 7683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:09.048336 7688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:09.048372 7689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:09.048408 7691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:09.049100 7683 server_base.cc:1034] running on GCE node
I20260619 15:06:09.049777 7683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:09.050843 7683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:09.052075 7683 hybrid_clock.cc:648] HybridClock initialized: now 1781881569052025 us; error 44 us; skew 500 ppm
I20260619 15:06:09.052403 7683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:09 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881569, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:09.055619 7683 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:06:09.057070 7683 webserver.cc:460] Webserver started at http://127.24.103.66:35005/ using document root <none> and password file <none>
I20260619 15:06:09.057790 7683 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:09.057945 7683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:09.063455 7683 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:06:09.066296 7698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:09.067720 7683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 15:06:09.067911 7683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "d416e496633a4b4ca187f7b0537adcca"
format_stamp: "Formatted at 2026-06-19 15:06:03 on dist-test-slave-zpbw"
server_key: "d227d62a65da6cfa1175ea6a2df88d36"
server_key_iv: "8860e2e2ce256bf49bb5a6ef75de022a"
server_key_version: "encryptionkey@0"
I20260619 15:06:09.068953 7683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:09.101536 7683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:09.105859 7683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:09.106159 7683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:09.107235 7683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:09.108651 7683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:09.108793 7683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:09.108924 7683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:09.109017 7683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:09.143010 7683 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:35171
I20260619 15:06:09.143071 7810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:35171 every 8 connection(s)
I20260619 15:06:09.144531 7683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:06:09.149423 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:09.147595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:38193)
0619 15:06:09.148319 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:09.148556 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":290,"threads_started":1}
W20260619 15:06:09.150096 7811 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:44191 (0 consecutive failures): Network error: Failed to ping master at 127.24.103.126:44191: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
I20260619 15:06:09.151034 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:09.150641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:55109)
0619 15:06:09.150841 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:09.150928 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 15:06:09.151927 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:09.151599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:43951)
0619 15:06:09.151723 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:09.151798 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 15:06:09.152225 7811 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 15:06:09.152417 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 7683
I20260619 15:06:09.153048 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:41901
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:44191
--webserver_port=43317
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:09.486032 7814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:09.486335 7814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:09.486503 7814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:09.494649 7814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:09.494822 7814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:09.495035 7814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:06:09.505486 7814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:41901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=43317
--tserver_master_addrs=127.24.103.126:44191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:09.506915 7814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:09.508240 7814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:09.521926 7820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:09.521847 7822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:09.521958 7819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:09.523037 7814 server_base.cc:1034] running on GCE node
I20260619 15:06:09.523703 7814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:09.524791 7814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:09.526050 7814 hybrid_clock.cc:648] HybridClock initialized: now 1781881569526004 us; error 35 us; skew 500 ppm
I20260619 15:06:09.526404 7814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:09 dist-test-slave-zpbw krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881569, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:09.529675 7814 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:06:09.531241 7814 webserver.cc:460] Webserver started at http://127.24.103.67:43317/ using document root <none> and password file <none>
I20260619 15:06:09.531966 7814 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:09.532121 7814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:09.537645 7814 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 15:06:09.540617 7829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:09.542016 7814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 15:06:09.542213 7814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "f1aea936947e41709523a572b8836648"
format_stamp: "Formatted at 2026-06-19 15:06:03 on dist-test-slave-zpbw"
server_key: "14b75e4533e26ac3450a5cff77dc6dbb"
server_key_iv: "20d5b5ff8b38b3f1b32e4e145174970f"
server_key_version: "encryptionkey@0"
I20260619 15:06:09.543337 7814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:09.589015 7814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:09.593338 7814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:09.593665 7814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:09.594681 7814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:09.596133 7814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:09.596266 7814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:09.596408 7814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:09.596503 7814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:09.630702 7814 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:41901
I20260619 15:06:09.630844 7941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:41901 every 8 connection(s)
I20260619 15:06:09.632229 7814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:06:09.637295 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 7814
I20260619 15:06:09.637861 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:09.635614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:42519)
0619 15:06:09.636348 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:09.636605 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":288,"threads_started":1}
W20260619 15:06:09.638563 7942 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:44191 (0 consecutive failures): Network error: Failed to ping master at 127.24.103.126:44191: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
I20260619 15:06:09.639564 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:09.639154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:47679)
0619 15:06:09.639352 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:09.639444 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 15:06:09.640550 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:09.640211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:55819)
0619 15:06:09.640353 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:09.640442 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 15:06:09.640836 7942 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 15:06:09.677191 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:09.676407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:45035)
0619 15:06:09.676841 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:09.676959 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":210,"threads_started":1}
I20260619 15:06:10.153590 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:10.152761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:42953)
0619 15:06:10.153217 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:10.153353 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":225,"threads_started":1}
I20260619 15:06:10.642865 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:10.641776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:49531)
0619 15:06:10.642437 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:10.642576 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":278,"threads_started":1}
I20260619 15:06:10.679164 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:10.678138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:34515)
0619 15:06:10.678692 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:10.678841 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":250,"threads_started":1}
I20260619 15:06:11.155419 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:11.154557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:53229)
0619 15:06:11.155154 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:11.155244 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":310,"threads_started":1}
I20260619 15:06:11.644610 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:11.643854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:60861)
0619 15:06:11.644346 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:11.644446 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":239,"threads_started":1}
I20260619 15:06:11.681042 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:11.680156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:51343)
0619 15:06:11.680723 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:11.680834 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":338,"threads_started":1}
I20260619 15:06:12.157131 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:12.156432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:57059)
0619 15:06:12.156873 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:12.156961 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260619 15:06:12.646704 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:12.645779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:40169)
0619 15:06:12.646319 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:12.646433 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":299,"threads_started":1}
I20260619 15:06:12.682978 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:12.682084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:57839)
0619 15:06:12.682655 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:12.682762 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":293,"threads_started":1}
I20260619 15:06:13.159242 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:13.158243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:45917)
0619 15:06:13.158854 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:13.158984 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":290,"threads_started":1}
I20260619 15:06:13.648330 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:13.647655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:49221)
0619 15:06:13.648085 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:13.648168 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":212,"threads_started":1}
I20260619 15:06:13.684659 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:13.683914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:53351)
0619 15:06:13.684384 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:13.684487 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":232,"threads_started":1}
I20260619 15:06:14.161082 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:14.160278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:47815)
0619 15:06:14.160787 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:14.160896 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
I20260619 15:06:14.650120 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:14.649358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:59907)
0619 15:06:14.649854 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:14.649945 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":253,"threads_started":1}
I20260619 15:06:14.686450 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:14.685647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:49527)
0619 15:06:14.686181 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:14.686287 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":275,"threads_started":1}
I20260619 15:06:15.162887 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:15.162077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:44751)
0619 15:06:15.162569 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:15.162683 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":254,"threads_started":1}
I20260619 15:06:15.651831 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:15.651082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:36523)
0619 15:06:15.651558 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:15.651655 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":261,"threads_started":1}
I20260619 15:06:15.688470 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:15.687531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:60209)
0619 15:06:15.688120 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:15.688224 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":286,"threads_started":1}
I20260619 15:06:16.164623 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:16.163793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:33163)
0619 15:06:16.164297 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:16.164408 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":227,"threads_started":1}
I20260619 15:06:16.653661 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:16.652820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:40821)
0619 15:06:16.653321 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:16.653430 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":219,"threads_started":1}
I20260619 15:06:16.690258 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:16.689458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:34159)
0619 15:06:16.689966 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:16.690062 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":245,"threads_started":1}
I20260619 15:06:17.166303 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:17.165582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:58571)
0619 15:06:17.166042 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:17.166130 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":226,"threads_started":1}
I20260619 15:06:17.655431 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:17.654671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:40511)
0619 15:06:17.655173 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:17.655260 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":278,"threads_started":1}
I20260619 15:06:17.692203 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:17.691373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:38587)
0619 15:06:17.691929 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:17.692023 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":276,"threads_started":1}
I20260619 15:06:18.168216 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:18.167427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:41143)
0619 15:06:18.167916 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:18.168049 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":229,"threads_started":1}
I20260619 15:06:18.657141 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:18.656350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:39203)
0619 15:06:18.656849 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:18.656952 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":242,"threads_started":1}
I20260619 15:06:18.694136 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:18.693367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:39447)
0619 15:06:18.693873 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:18.693969 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":290,"threads_started":1}
I20260619 15:06:19.170063 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:19.169345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:42357)
0619 15:06:19.169792 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:19.169894 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":238,"threads_started":1}
I20260619 15:06:19.658985 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:19.658265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:44273)
0619 15:06:19.658710 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:19.658825 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":229,"threads_started":1}
I20260619 15:06:19.695904 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:19.695067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:41317)
0619 15:06:19.695591 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:19.695690 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":295,"threads_started":1}
I20260619 15:06:20.171868 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:20.171179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:60635)
0619 15:06:20.171600 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:20.171692 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":203,"threads_started":1}
I20260619 15:06:20.660761 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:20.660007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:46317)
0619 15:06:20.660471 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:20.660578 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":221,"threads_started":1}
I20260619 15:06:20.697857 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:20.696929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:38253)
0619 15:06:20.697487 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:20.697620 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":224,"threads_started":1}
I20260619 15:06:21.173738 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:21.172908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:53557)
0619 15:06:21.173426 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:21.173516 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":226,"threads_started":1}
I20260619 15:06:21.662664 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:21.661930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:58901)
0619 15:06:21.662419 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:21.662509 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":247,"threads_started":1}
I20260619 15:06:21.699831 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:21.699020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:40119)
0619 15:06:21.699535 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:21.699639 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":237,"threads_started":1}
I20260619 15:06:22.175608 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:22.174731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:33157)
0619 15:06:22.175293 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:22.175419 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":262,"threads_started":1}
I20260619 15:06:22.664513 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:22.663681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:39305)
0619 15:06:22.664201 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:22.664303 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":263,"threads_started":1}
I20260619 15:06:22.701740 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:22.700884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:40547)
0619 15:06:22.701416 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:22.701564 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":294,"threads_started":1}
I20260619 15:06:23.177297 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:23.176587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:38037)
0619 15:06:23.177026 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:23.177116 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":211,"threads_started":1}
I20260619 15:06:23.666464 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:23.665517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:47465)
0619 15:06:23.666074 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:23.666199 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":286,"threads_started":1}
I20260619 15:06:23.703855 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:23.703057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:41287)
0619 15:06:23.703526 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:23.703660 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":226,"threads_started":1}
I20260619 15:06:24.179332 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:24.178557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.66:34431)
0619 15:06:24.179053 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:24.179151 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":282,"threads_started":1}
I20260619 15:06:24.640826 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 7520
I20260619 15:06:24.659956 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:24.642286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.24.103.65:40079)
0619 15:06:24.642955 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:24.642970 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:24.643016 (+ 46us) server_negotiation.cc:407] Connection header received
0619 15:06:24.644600 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:24.644610 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:24.645067 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:24.645374 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:24.647097 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:24.648528 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:24.649831 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:24.650544 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:24.650685 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:24.653649 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:24.653656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:24.653673 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:24.653717 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:24.656752 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:24.657493 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:24.657500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:24.657506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:24.657689 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:24.658066 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:24.658069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:24.658071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:24.658628 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:24.658951 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:24.659322 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 15:06:24.659669 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":324,"threads_started":1}
I20260619 15:06:24.661516 7675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:24.668496 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:24.667663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.67:53979)
0619 15:06:24.668186 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:24.668308 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":285,"threads_started":1}
I20260619 15:06:24.752139 7993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:24.746819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44191 (local address 127.24.103.65:37061)
0619 15:06:24.751657 (+ 4838us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:24.751886 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4614,"thread_start_us":4472,"threads_started":1}
I20260619 15:06:24.752820 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 7520
I20260619 15:06:24.770315 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 7683
I20260619 15:06:24.789216 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:24.771429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.24.103.66:35171)
0619 15:06:24.772061 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:24.772075 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:24.772120 (+ 45us) server_negotiation.cc:407] Connection header received
0619 15:06:24.773946 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:24.773962 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:24.774455 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:24.774768 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:24.776242 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:24.777508 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:24.778865 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:24.779578 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:24.779720 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:24.782505 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:24.782514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:24.782531 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:24.782571 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:24.785879 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:24.786602 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:24.786609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:24.786616 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:24.786849 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:24.787278 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:24.787281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:24.787283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:24.787833 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:24.788070 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:24.788456 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 15:06:24.788913 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":307,"threads_started":1}
I20260619 15:06:24.790647 7806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:24.876827 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 7683
I20260619 15:06:24.898444 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 7814
I20260619 15:06:24.917857 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:24.899675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.24.103.67:41901)
0619 15:06:24.900307 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:24.900320 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:24.900362 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:06:24.902023 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:24.902033 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:24.902503 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:24.902938 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:24.904556 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:24.905805 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:24.907231 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:24.907942 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:24.908088 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:24.910982 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:24.910991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:24.911008 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:24.911051 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:24.914221 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:24.915308 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:24.915315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:24.915322 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:24.915534 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:24.915881 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:24.915884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:24.915886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:24.916467 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:24.916726 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:24.917089 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 15:06:24.917561 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":328,"threads_started":1}
I20260619 15:06:24.919368 7937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:25.005079 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 7814
2026-06-19T15:06:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24719 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:06:25 dist-test-slave-zpbw krb5kdc[8000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:06:25 dist-test-slave-zpbw krb5kdc[8000](info): set up 2 sockets
Jun 19 15:06:25 dist-test-slave-zpbw krb5kdc[8000](info): commencing operation
krb5kdc: starting...
W20260619 15:06:26.103823 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:06:26 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:06:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:06:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:06:26.352675 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34375
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:34375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:26.695806 8013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:26.696177 8013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:26.696360 8013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:26.703765 8013 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 15:06:26.703922 8013 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 15:06:26.704016 8013 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 15:06:26.704113 8013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:06:26.704216 8013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:26.704321 8013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:26.704429 8013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:06:26.704535 8013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:06:26.714900 8013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:34375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:26.716360 8013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:26.717761 8013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:26.730077 8018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:26.730186 8021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:26.730139 8019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:26.731110 8013 server_base.cc:1034] running on GCE node
I20260619 15:06:26.732007 8013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:26.733590 8013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:26.734901 8013 hybrid_clock.cc:648] HybridClock initialized: now 1781881586734826 us; error 39 us; skew 500 ppm
I20260619 15:06:26.735270 8013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:26 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881586, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:26.742506 8013 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:06:26.744347 8013 webserver.cc:460] Webserver started at http://127.24.103.126:41319/ using document root <none> and password file <none>
I20260619 15:06:26.745126 8013 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:26.745265 8013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:26.745609 8013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:06:26.748893 8013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "e0f338b8b9c442269a35a15e86599a23"
format_stamp: "Formatted at 2026-06-19 15:06:26 on dist-test-slave-zpbw"
server_key: "ee169b68ebb9938aa69b5d7c5ab04da5"
server_key_iv: "1d74969d1a430f3ac1602eeec813fb93"
server_key_version: "encryptionkey@0"
I20260619 15:06:26.749765 8013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "e0f338b8b9c442269a35a15e86599a23"
format_stamp: "Formatted at 2026-06-19 15:06:26 on dist-test-slave-zpbw"
server_key: "ee169b68ebb9938aa69b5d7c5ab04da5"
server_key_iv: "1d74969d1a430f3ac1602eeec813fb93"
server_key_version: "encryptionkey@0"
I20260619 15:06:26.755486 8013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 15:06:26.759328 8028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:26.760831 8013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:06:26.761030 8013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "e0f338b8b9c442269a35a15e86599a23"
format_stamp: "Formatted at 2026-06-19 15:06:26 on dist-test-slave-zpbw"
server_key: "ee169b68ebb9938aa69b5d7c5ab04da5"
server_key_iv: "1d74969d1a430f3ac1602eeec813fb93"
server_key_version: "encryptionkey@0"
I20260619 15:06:26.761276 8013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:26.781186 8013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:26.785722 8013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:26.786015 8013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:26.805647 8013 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:34375
I20260619 15:06:26.805627 8079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:34375 every 8 connection(s)
I20260619 15:06:26.807226 8013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:06:26.807864 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 8013
I20260619 15:06:26.808178 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:06:26.809054 24989 external_mini_cluster.cc:1428] Setting key c43cb142c193b9a08cb17756709a678f
I20260619 15:06:26.811913 8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:26.823096 8080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Bootstrap starting.
Jun 19 15:06:26 dist-test-slave-zpbw krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:26.826462 8080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:26.827670 8080 log.cc:824] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:26.831218 8080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: No bootstrap required, opened a new log
I20260619 15:06:26.832777 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:26.811268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.24.103.126:34375)
0619 15:06:26.812195 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:26.812208 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:26.812255 (+ 47us) server_negotiation.cc:407] Connection header received
0619 15:06:26.813909 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:26.813945 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:26.814351 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:26.814792 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:26.816441 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:26.817808 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:26.819235 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:26.819953 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:26.820170 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:26.824446 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:26.824457 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:26.824472 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:26.824516 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:26.827864 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:26.828895 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:26.828902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:26.828907 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:26.829164 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:26.829581 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:26.829589 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:26.829594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:26.830312 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:26.830609 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:26.830968 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 15:06:26.831411 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":415,"threads_started":1}
I20260619 15:06:26.840353 8080 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } }
I20260619 15:06:26.840745 8080 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:26.840883 8080 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f338b8b9c442269a35a15e86599a23, State: Initialized, Role: FOLLOWER
I20260619 15:06:26.841576 8080 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } }
I20260619 15:06:26.841969 8080 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:06:26.842130 8080 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:06:26.842311 8080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:26.844841 8080 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } }
I20260619 15:06:26.845392 8080 leader_election.cc:302] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0f338b8b9c442269a35a15e86599a23; no voters:
I20260619 15:06:26.846082 8080 leader_election.cc:288] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:06:26.846272 8085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:06:26.847159 8085 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 1 LEADER]: Becoming Leader. State: Replica: e0f338b8b9c442269a35a15e86599a23, State: Running, Role: LEADER
I20260619 15:06:26.847855 8085 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } }
I20260619 15:06:26.848479 8080 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:06:26.851187 8086 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0f338b8b9c442269a35a15e86599a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } } }
I20260619 15:06:26.851720 8086 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: This master's current role is: LEADER
I20260619 15:06:26.852277 8087 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f338b8b9c442269a35a15e86599a23. Latest consensus state: current_term: 1 leader_uuid: "e0f338b8b9c442269a35a15e86599a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } } }
I20260619 15:06:26.852628 8087 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: This master's current role is: LEADER
I20260619 15:06:26.853466 8092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:06:26.860394 8092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:06:26.870447 8092 catalog_manager.cc:1347] Generated new cluster ID: 47a08e463fde4fa683ded23334679d38
I20260619 15:06:26.870594 8092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:06:26.886202 8092 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:06:26.888305 8092 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:06:26.897614 8092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Generated new TSK 0
I20260619 15:06:26.898808 8092 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:06:26.972795 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34375
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:27.322850 8106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:27.323160 8106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:27.323329 8106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:27.331252 8106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:27.331370 8106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:27.331561 8106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:06:27.341976 8106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:27.343489 8106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:27.344946 8106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:27.359275 8111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:27.359565 8112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:27.359537 8114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:27.360180 8106 server_base.cc:1034] running on GCE node
I20260619 15:06:27.360836 8106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:27.361961 8106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:27.363197 8106 hybrid_clock.cc:648] HybridClock initialized: now 1781881587363144 us; error 49 us; skew 500 ppm
I20260619 15:06:27.363549 8106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:27 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881587, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:27.367283 8106 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:06:27.369027 8106 webserver.cc:460] Webserver started at http://127.24.103.65:36543/ using document root <none> and password file <none>
I20260619 15:06:27.369880 8106 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:27.370033 8106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:27.370388 8106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:06:27.373608 8106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "70664d89ba3545e5bbcf62a84f8c49b9"
format_stamp: "Formatted at 2026-06-19 15:06:27 on dist-test-slave-zpbw"
server_key: "86093e31dd646081e00156c208f3afbe"
server_key_iv: "48b4b49e6162dfd9c6fd5b6363022554"
server_key_version: "encryptionkey@0"
I20260619 15:06:27.374504 8106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "70664d89ba3545e5bbcf62a84f8c49b9"
format_stamp: "Formatted at 2026-06-19 15:06:27 on dist-test-slave-zpbw"
server_key: "86093e31dd646081e00156c208f3afbe"
server_key_iv: "48b4b49e6162dfd9c6fd5b6363022554"
server_key_version: "encryptionkey@0"
I20260619 15:06:27.380417 8106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 15:06:27.384104 8121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:27.385572 8106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:06:27.385793 8106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "70664d89ba3545e5bbcf62a84f8c49b9"
format_stamp: "Formatted at 2026-06-19 15:06:27 on dist-test-slave-zpbw"
server_key: "86093e31dd646081e00156c208f3afbe"
server_key_iv: "48b4b49e6162dfd9c6fd5b6363022554"
server_key_version: "encryptionkey@0"
I20260619 15:06:27.386039 8106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:27.422159 8106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:27.426756 8106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:27.427160 8106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:27.428262 8106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:27.429792 8106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:27.429908 8106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:27.430076 8106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:27.430155 8106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:27.465112 8106 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:34271
I20260619 15:06:27.465157 8233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:34271 every 8 connection(s)
I20260619 15:06:27.466609 8106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:06:27.471951 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 8106
I20260619 15:06:27.472203 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:06:27.472904 24989 external_mini_cluster.cc:1428] Setting key ac23141bf74e4aabca2b7ce822d98594
Jun 19 15:06:27 dist-test-slave-zpbw krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881587, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:27.490124 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:27.469253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:40259 (local address 127.24.103.126:34375)
0619 15:06:27.469667 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:27.469672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:27.470793 (+ 1121us) server_negotiation.cc:407] Connection header received
0619 15:06:27.472777 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:27.472785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:27.472870 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:27.473007 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:27.474725 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:27.475846 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:27.477491 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:27.478213 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:27.478351 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:27.482434 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:27.482444 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:27.482450 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:27.482499 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:27.486302 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:27.487227 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:27.487236 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:27.487242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:27.487401 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:27.487848 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:27.487854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:27.487857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:27.488263 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:27.488448 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:27.489307 (+ 859us) server_negotiation.cc:299] Negotiation successful
0619 15:06:27.489583 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":199,"threads_started":1}
I20260619 15:06:27.491039 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:27.469867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:40259)
0619 15:06:27.470589 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:27.470633 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:27.472492 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:27.473240 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:27.473249 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:27.473946 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:06:27.474541 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:27.474558 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:27.475998 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:27.476002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:27.477320 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:27.477333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:27.478443 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:27.478448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:27.478654 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:27.479442 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:06:27.479471 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:06:27.482127 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:06:27.486500 (+ 4373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:27.486508 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:27.486523 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:27.487089 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:27.487525 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:27.487529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:27.487531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:27.487718 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:27.488523 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:06:27.488529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:06:27.488878 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 15:06:27.489124 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 15:06:27.489563 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":291,"threads_started":1}
I20260619 15:06:27.492677 8234 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34375
I20260619 15:06:27.493103 8234 heartbeater.cc:464] Registering TS with master...
I20260619 15:06:27.494113 8234 heartbeater.cc:511] Master 127.24.103.126:34375 requested a full tablet report, sending...
I20260619 15:06:27.497079 8045 ts_manager.cc:194] Registered new tserver with Master: 70664d89ba3545e5bbcf62a84f8c49b9 (127.24.103.65:34271)
I20260619 15:06:27.499368 8045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:40259
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:06:27.530133 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34375
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:27.875413 8240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:27.875725 8240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:27.875895 8240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:27.883714 8240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:27.883837 8240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:27.884013 8240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:06:27.894403 8240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:27.895848 8240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:27.897249 8240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:27.911058 8248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:27.911150 8246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:27.911121 8245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:27.912026 8240 server_base.cc:1034] running on GCE node
I20260619 15:06:27.912722 8240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:27.913902 8240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:27.915102 8240 hybrid_clock.cc:648] HybridClock initialized: now 1781881587915053 us; error 40 us; skew 500 ppm
I20260619 15:06:27.915432 8240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:27 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881587, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:27.918706 8240 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:06:27.920393 8240 webserver.cc:460] Webserver started at http://127.24.103.66:38551/ using document root <none> and password file <none>
I20260619 15:06:27.921135 8240 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:27.921291 8240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:27.921625 8240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:06:27.924799 8240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "403c75060d464222b2a712e98303e3d3"
format_stamp: "Formatted at 2026-06-19 15:06:27 on dist-test-slave-zpbw"
server_key: "ab8c58584ba836841ee0c9df5444b7f4"
server_key_iv: "70bc673bfa569ed8ebfdd17872d915a3"
server_key_version: "encryptionkey@0"
I20260619 15:06:27.925624 8240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "403c75060d464222b2a712e98303e3d3"
format_stamp: "Formatted at 2026-06-19 15:06:27 on dist-test-slave-zpbw"
server_key: "ab8c58584ba836841ee0c9df5444b7f4"
server_key_iv: "70bc673bfa569ed8ebfdd17872d915a3"
server_key_version: "encryptionkey@0"
I20260619 15:06:27.931372 8240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 15:06:27.935315 8255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:27.936762 8240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:06:27.936960 8240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "403c75060d464222b2a712e98303e3d3"
format_stamp: "Formatted at 2026-06-19 15:06:27 on dist-test-slave-zpbw"
server_key: "ab8c58584ba836841ee0c9df5444b7f4"
server_key_iv: "70bc673bfa569ed8ebfdd17872d915a3"
server_key_version: "encryptionkey@0"
I20260619 15:06:27.937182 8240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:27.952932 8240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:27.957886 8240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:27.958254 8240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:27.959372 8240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:27.960896 8240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:27.960981 8240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:27.961146 8240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:27.961203 8240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:27.995066 8240 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:43261
I20260619 15:06:27.995090 8367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:43261 every 8 connection(s)
I20260619 15:06:27.996541 8240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:06:28.005384 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 8240
I20260619 15:06:28.005622 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:06:28.006131 24989 external_mini_cluster.cc:1428] Setting key 81a6727261821cae34cae3f57e6e9dde
Jun 19 15:06:28 dist-test-slave-zpbw krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881587, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:28.019783 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:27.999295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:46959 (local address 127.24.103.126:34375)
0619 15:06:27.999766 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:27.999770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:28.000684 (+ 914us) server_negotiation.cc:407] Connection header received
0619 15:06:28.002547 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:28.002553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:28.002646 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:28.002769 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:28.004455 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.005405 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.007543 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.008079 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.008197 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.012197 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:28.012207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:28.012215 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:28.012267 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:28.016168 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:28.017299 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:28.017303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:28.017305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:28.017452 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:28.017981 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:28.017984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:28.017986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:28.018255 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:28.018388 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:28.019327 (+ 939us) server_negotiation.cc:299] Negotiation successful
0619 15:06:28.019527 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":262,"threads_started":1}
I20260619 15:06:28.021524 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:27.999766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:46959)
0619 15:06:28.000508 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:28.000554 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:28.002254 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:28.003082 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:28.003091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:28.003676 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:06:28.004267 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.004284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.005640 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.005645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.007391 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.007408 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.008353 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.008362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.008645 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.009392 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:06:28.009418 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:06:28.011892 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:06:28.016386 (+ 4494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:28.016400 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:28.016422 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:28.017148 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:28.017573 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:28.017581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:28.017587 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:28.017857 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:28.018384 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:06:28.018395 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:06:28.018946 (+ 551us) client_negotiation.cc:769] Sending connection context
0619 15:06:28.019248 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 15:06:28.019865 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":278,"threads_started":1}
I20260619 15:06:28.023155 8368 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34375
I20260619 15:06:28.023485 8368 heartbeater.cc:464] Registering TS with master...
I20260619 15:06:28.024262 8368 heartbeater.cc:511] Master 127.24.103.126:34375 requested a full tablet report, sending...
I20260619 15:06:28.025825 8045 ts_manager.cc:194] Registered new tserver with Master: 403c75060d464222b2a712e98303e3d3 (127.24.103.66:43261)
I20260619 15:06:28.026681 8045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:46959
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:06:28.061661 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34375
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:28.396770 8374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:28.397142 8374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:28.397308 8374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:28.404793 8374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:28.404974 8374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:28.405149 8374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:06:28.415162 8374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:28.416519 8374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:28.417873 8374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:28.431380 8382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:28.431474 8380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:28.431694 8379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:28.432368 8374 server_base.cc:1034] running on GCE node
I20260619 15:06:28.433055 8374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:28.434070 8374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:28.435303 8374 hybrid_clock.cc:648] HybridClock initialized: now 1781881588435242 us; error 61 us; skew 500 ppm
I20260619 15:06:28.435652 8374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:28 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881588, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:28.438982 8374 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:06:28.440541 8374 webserver.cc:460] Webserver started at http://127.24.103.67:39001/ using document root <none> and password file <none>
I20260619 15:06:28.441349 8374 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:28.441505 8374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:28.441890 8374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:06:28.445024 8374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "961f6fae5e564d9a90ef4ce887500379"
format_stamp: "Formatted at 2026-06-19 15:06:28 on dist-test-slave-zpbw"
server_key: "19673095930822a7026bf497f3b6df18"
server_key_iv: "5bb97b51dbd10eb9d45466db67a1a019"
server_key_version: "encryptionkey@0"
I20260619 15:06:28.445817 8374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "961f6fae5e564d9a90ef4ce887500379"
format_stamp: "Formatted at 2026-06-19 15:06:28 on dist-test-slave-zpbw"
server_key: "19673095930822a7026bf497f3b6df18"
server_key_iv: "5bb97b51dbd10eb9d45466db67a1a019"
server_key_version: "encryptionkey@0"
I20260619 15:06:28.451762 8374 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 15:06:28.455510 8389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:28.457023 8374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:06:28.457252 8374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "961f6fae5e564d9a90ef4ce887500379"
format_stamp: "Formatted at 2026-06-19 15:06:28 on dist-test-slave-zpbw"
server_key: "19673095930822a7026bf497f3b6df18"
server_key_iv: "5bb97b51dbd10eb9d45466db67a1a019"
server_key_version: "encryptionkey@0"
I20260619 15:06:28.457513 8374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:28.478169 8374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:28.482417 8374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:28.482745 8374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:28.483826 8374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:28.485229 8374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:28.485306 8374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:28.485426 8374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:28.485499 8374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:28.503043 8234 heartbeater.cc:503] Master 127.24.103.126:34375 was elected leader, sending a full tablet report...
I20260619 15:06:28.523196 8374 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:45535
I20260619 15:06:28.523298 8501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:45535 every 8 connection(s)
I20260619 15:06:28.524542 8374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:06:28.526391 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 8374
I20260619 15:06:28.526598 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:06:28.527215 24989 external_mini_cluster.cc:1428] Setting key 334d1abfb922088d2841debdd99cf532
Jun 19 15:06:28 dist-test-slave-zpbw krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881588, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:28.546833 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.527466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:40115 (local address 127.24.103.126:34375)
0619 15:06:28.527920 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:28.527924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:28.529242 (+ 1318us) server_negotiation.cc:407] Connection header received
0619 15:06:28.531433 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:28.531440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:28.531520 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:28.531629 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:28.533165 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.534024 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.535858 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.536248 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.536352 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.539870 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:28.539875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:28.539879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:28.539915 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:28.543504 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:28.544326 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:28.544331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:28.544335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:28.544472 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:28.544942 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:28.544947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:28.544950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:28.545234 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:28.545406 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:28.546138 (+ 732us) server_negotiation.cc:299] Negotiation successful
0619 15:06:28.546370 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":234,"threads_started":1}
I20260619 15:06:28.547945 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.528127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:40115)
0619 15:06:28.529037 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:28.529103 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:28.531185 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:28.531849 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:28.531858 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:28.532420 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:06:28.533012 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.533029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.534271 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.534276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.535717 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.535729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.536446 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.536450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.536695 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.537335 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:06:28.537363 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:06:28.539640 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:06:28.543668 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:28.543679 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:28.543699 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:28.544189 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:28.544604 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:28.544610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:28.544615 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:28.544814 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:28.545374 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:06:28.545382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:06:28.545768 (+ 386us) client_negotiation.cc:769] Sending connection context
0619 15:06:28.546047 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 15:06:28.546653 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":321,"threads_started":1}
I20260619 15:06:28.549515 8502 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34375
I20260619 15:06:28.549818 8502 heartbeater.cc:464] Registering TS with master...
I20260619 15:06:28.550551 8502 heartbeater.cc:511] Master 127.24.103.126:34375 requested a full tablet report, sending...
I20260619 15:06:28.552062 8045 ts_manager.cc:194] Registered new tserver with Master: 961f6fae5e564d9a90ef4ce887500379 (127.24.103.67:45535)
I20260619 15:06:28.552857 8045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:40115
I20260619 15:06:28.556062 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:06:28.574294 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.560835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.24.103.126:34375)
0619 15:06:28.561108 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:28.561115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:28.561478 (+ 363us) server_negotiation.cc:407] Connection header received
0619 15:06:28.561715 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:28.561722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:28.561844 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:28.561979 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:28.563661 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.564437 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.565797 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.566334 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.566441 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.568801 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:28.568806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:28.568810 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:28.568846 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:28.571816 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:28.572515 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:28.572519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:28.572521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:28.572653 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:28.572984 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:28.572987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:28.572989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:28.573358 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:28.573581 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:28.573852 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 15:06:28.574040 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 15:06:28.578588 8045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:06:28.585760 8045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:06:28.612063 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.599791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:41192)
0619 15:06:28.602464 (+ 2673us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:28.602485 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:28.602605 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:28.604801 (+ 2196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:28.604806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:28.604835 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:28.605486 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.605501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.606947 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.606951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.609281 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.609293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.610943 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.610950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.611141 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.611202 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 15:06:28.611356 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 15:06:28.611568 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2528,"mutex_wait_us":221,"thread_start_us":3093,"threads_started":1}
I20260619 15:06:28.612195 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.600109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.24.103.65:34271)
0619 15:06:28.604204 (+ 4095us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:28.604212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:28.604239 (+ 27us) server_negotiation.cc:407] Connection header received
0619 15:06:28.604336 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:28.604343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:28.604574 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:28.604786 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:28.605693 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.606731 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.609456 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.610686 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.610862 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.611121 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:28.611608 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 15:06:28.611814 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3909,"thread_start_us":4015,"threads_started":1}
I20260619 15:06:28.612639 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.599609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:43261 (local address 127.0.0.1:60372)
0619 15:06:28.601524 (+ 1915us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:28.601583 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:28.601809 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:28.604568 (+ 2759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:28.604575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:28.604634 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:28.605104 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.605116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.607365 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.607371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.609870 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.609889 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.611755 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.611762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.611922 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.611949 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 15:06:28.612018 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 15:06:28.612148 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1577,"mutex_wait_us":1,"thread_start_us":1398,"threads_started":1}
I20260619 15:06:28.613552 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.599805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.24.103.66:43261)
0619 15:06:28.603209 (+ 3404us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:28.603220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:28.603253 (+ 33us) server_negotiation.cc:407] Connection header received
0619 15:06:28.603363 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:28.603371 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:28.603630 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:28.605259 (+ 1629us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:28.605516 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.607197 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.610043 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.611482 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.612658 (+ 1176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.612985 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:28.613062 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 15:06:28.613224 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3252,"thread_start_us":3139,"threads_started":1}
I20260619 15:06:28.613562 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.602684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.24.103.67:45535)
0619 15:06:28.603788 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:28.603799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:28.604397 (+ 598us) server_negotiation.cc:407] Connection header received
0619 15:06:28.604512 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:28.604521 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:28.604735 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:28.604954 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:28.605795 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.607365 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.610095 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.611142 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.611273 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.611493 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:28.613123 (+ 1630us) server_negotiation.cc:299] Negotiation successful
0619 15:06:28.613284 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":631,"threads_started":1}
I20260619 15:06:28.614097 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.602040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:45535 (local address 127.0.0.1:43564)
0619 15:06:28.604243 (+ 2203us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:28.604260 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:28.604368 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:28.605061 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:28.605065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:28.605099 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:28.605577 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.605586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.607533 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.607539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.609929 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.609942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.611356 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.611363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.611530 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.611576 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 15:06:28.611665 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 15:06:28.613730 (+ 2065us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2095,"mutex_wait_us":97,"thread_start_us":1932,"threads_started":1}
I20260619 15:06:28.616801 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 641fd7dddc4c4713a7fab9cda9ce1c6f (DEFAULT_TABLE table=test-table [id=cf0999eacce5482f8b1b8a9ef7832d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:06:28.617076 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 641fd7dddc4c4713a7fab9cda9ce1c6f (DEFAULT_TABLE table=test-table [id=cf0999eacce5482f8b1b8a9ef7832d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:06:28.616984 8437 tablet_service.cc:1467] Processing CreateTablet for tablet 641fd7dddc4c4713a7fab9cda9ce1c6f (DEFAULT_TABLE table=test-table [id=cf0999eacce5482f8b1b8a9ef7832d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:06:28.618378 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 641fd7dddc4c4713a7fab9cda9ce1c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:28.618450 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 641fd7dddc4c4713a7fab9cda9ce1c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:28.618449 8437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 641fd7dddc4c4713a7fab9cda9ce1c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:28.631062 8521 tablet_bootstrap.cc:492] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9: Bootstrap starting.
I20260619 15:06:28.634474 8521 tablet_bootstrap.cc:654] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:28.635722 8521 log.cc:824] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:28.636539 8523 tablet_bootstrap.cc:492] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3: Bootstrap starting.
I20260619 15:06:28.637547 8522 tablet_bootstrap.cc:492] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379: Bootstrap starting.
I20260619 15:06:28.640777 8523 tablet_bootstrap.cc:654] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:28.641084 8522 tablet_bootstrap.cc:654] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:28.641664 8521 tablet_bootstrap.cc:492] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9: No bootstrap required, opened a new log
I20260619 15:06:28.641945 8521 ts_tablet_manager.cc:1397] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260619 15:06:28.642462 8523 log.cc:824] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:28.642717 8522 log.cc:824] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:28.645761 8522 tablet_bootstrap.cc:492] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379: No bootstrap required, opened a new log
I20260619 15:06:28.645759 8523 tablet_bootstrap.cc:492] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3: No bootstrap required, opened a new log
I20260619 15:06:28.646049 8522 ts_tablet_manager.cc:1397] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 15:06:28.646063 8523 ts_tablet_manager.cc:1397] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 15:06:28.650435 8521 raft_consensus.cc:348] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } } peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } }
I20260619 15:06:28.650875 8521 raft_consensus.cc:374] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:28.650992 8521 raft_consensus.cc:729] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70664d89ba3545e5bbcf62a84f8c49b9, State: Initialized, Role: FOLLOWER
I20260619 15:06:28.651674 8521 consensus_queue.cc:260] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } } peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } }
I20260619 15:06:28.656415 8521 ts_tablet_manager.cc:1428] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260619 15:06:28.657402 8522 raft_consensus.cc:348] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } } peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } }
I20260619 15:06:28.657403 8523 raft_consensus.cc:348] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } } peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } }
I20260619 15:06:28.657930 8523 raft_consensus.cc:374] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:28.657924 8522 raft_consensus.cc:374] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:28.658038 8523 raft_consensus.cc:729] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403c75060d464222b2a712e98303e3d3, State: Initialized, Role: FOLLOWER
I20260619 15:06:28.658038 8522 raft_consensus.cc:729] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 961f6fae5e564d9a90ef4ce887500379, State: Initialized, Role: FOLLOWER
I20260619 15:06:28.658702 8522 consensus_queue.cc:260] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } } peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } }
I20260619 15:06:28.658702 8523 consensus_queue.cc:260] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } } peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } }
I20260619 15:06:28.660104 8368 heartbeater.cc:503] Master 127.24.103.126:34375 was elected leader, sending a full tablet report...
I20260619 15:06:28.660472 8502 heartbeater.cc:503] Master 127.24.103.126:34375 was elected leader, sending a full tablet report...
I20260619 15:06:28.660761 8523 ts_tablet_manager.cc:1428] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260619 15:06:28.661008 8522 ts_tablet_manager.cc:1428] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260619 15:06:28.719277 8235 tablet.cc:2369] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:06:28.748812 8369 tablet.cc:2369] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:06:28.776206 8503 tablet.cc:2369] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:06:28.813679 8527 raft_consensus.cc:482] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:06:28.814045 8527 raft_consensus.cc:504] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } } peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } }
I20260619 15:06:28.815981 8527 leader_election.cc:288] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 403c75060d464222b2a712e98303e3d3 (127.24.103.66:43261), 961f6fae5e564d9a90ef4ce887500379 (127.24.103.67:45535)
I20260619 15:06:28.823884 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.816477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:39067 (local address 127.24.103.66:43261)
0619 15:06:28.816746 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:28.816757 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:28.817399 (+ 642us) server_negotiation.cc:407] Connection header received
0619 15:06:28.817724 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:28.817735 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:28.817894 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:28.817982 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:28.818919 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.819825 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.821792 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.822764 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.822951 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.823010 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:28.823438 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 15:06:28.823594 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 15:06:28.823961 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.817604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:59715 (local address 127.24.103.67:45535)
0619 15:06:28.817780 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:28.817786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:28.818401 (+ 615us) server_negotiation.cc:407] Connection header received
0619 15:06:28.818572 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:28.818578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:28.818663 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:28.818763 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:28.819384 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.820276 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.822011 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.823013 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.823171 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.823244 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:28.823562 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 15:06:28.823714 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 15:06:28.824156 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.816400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:43261 (local address 127.24.103.65:39067)
0619 15:06:28.817302 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:28.817324 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:28.817515 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:28.818190 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:28.818193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:28.818223 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:28.818683 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.818698 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.819969 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.819975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.821599 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.821610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.823036 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.823044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.823237 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.823280 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 15:06:28.823368 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 15:06:28.823504 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":1,"spinlock_wait_cycles":44928,"thread_start_us":576,"threads_started":1}
I20260619 15:06:28.824290 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.817476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:45535 (local address 127.24.103.65:59715)
0619 15:06:28.818247 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:28.818264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:28.818425 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:28.818816 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:28.818823 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:28.818853 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:28.819228 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.819241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.820411 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.820415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.821859 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:28.821871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.823216 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.823220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:28.823382 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.823417 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 15:06:28.823511 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 15:06:28.823630 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":446,"threads_started":1}
I20260619 15:06:28.825167 8457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641fd7dddc4c4713a7fab9cda9ce1c6f" candidate_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "961f6fae5e564d9a90ef4ce887500379" is_pre_election: true
I20260619 15:06:28.825165 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641fd7dddc4c4713a7fab9cda9ce1c6f" candidate_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403c75060d464222b2a712e98303e3d3" is_pre_election: true
I20260619 15:06:28.825706 8457 raft_consensus.cc:2454] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70664d89ba3545e5bbcf62a84f8c49b9 in term 0.
I20260619 15:06:28.825706 8323 raft_consensus.cc:2454] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70664d89ba3545e5bbcf62a84f8c49b9 in term 0.
I20260619 15:06:28.826390 8123 leader_election.cc:302] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70664d89ba3545e5bbcf62a84f8c49b9, 961f6fae5e564d9a90ef4ce887500379; no voters:
I20260619 15:06:28.826856 8527 raft_consensus.cc:2781] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:06:28.827019 8527 raft_consensus.cc:482] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:06:28.827162 8527 raft_consensus.cc:3037] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:28.829437 8527 raft_consensus.cc:504] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } } peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } }
I20260619 15:06:28.830159 8527 leader_election.cc:288] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [CANDIDATE]: Term 1 election: Requested vote from peers 403c75060d464222b2a712e98303e3d3 (127.24.103.66:43261), 961f6fae5e564d9a90ef4ce887500379 (127.24.103.67:45535)
I20260619 15:06:28.830652 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641fd7dddc4c4713a7fab9cda9ce1c6f" candidate_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403c75060d464222b2a712e98303e3d3"
I20260619 15:06:28.830660 8457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641fd7dddc4c4713a7fab9cda9ce1c6f" candidate_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "961f6fae5e564d9a90ef4ce887500379"
I20260619 15:06:28.830891 8323 raft_consensus.cc:3037] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:28.830891 8457 raft_consensus.cc:3037] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:28.834237 8323 raft_consensus.cc:2454] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70664d89ba3545e5bbcf62a84f8c49b9 in term 1.
I20260619 15:06:28.834237 8457 raft_consensus.cc:2454] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70664d89ba3545e5bbcf62a84f8c49b9 in term 1.
I20260619 15:06:28.834704 8123 leader_election.cc:302] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70664d89ba3545e5bbcf62a84f8c49b9, 961f6fae5e564d9a90ef4ce887500379; no voters:
I20260619 15:06:28.835094 8527 raft_consensus.cc:2781] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:06:28.835557 8527 raft_consensus.cc:686] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 1 LEADER]: Becoming Leader. State: Replica: 70664d89ba3545e5bbcf62a84f8c49b9, State: Running, Role: LEADER
I20260619 15:06:28.836040 8527 consensus_queue.cc:237] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } } peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } }
I20260619 15:06:28.840782 8044 catalog_manager.cc:5495] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70664d89ba3545e5bbcf62a84f8c49b9 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } health_report { overall_health: UNKNOWN } } }
I20260619 15:06:28.909351 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.904063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.24.103.65:34271)
0619 15:06:28.904250 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:28.904254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:28.904272 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:06:28.904449 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:28.904452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:28.904532 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:28.904666 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:06:28.905012 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.905731 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.907229 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.907768 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.907867 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.908354 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:06:28.908860 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:06:28.908967 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:28.909064 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 15:06:28.909133 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 15:06:28.917657 8323 raft_consensus.cc:1261] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3 [term 1 FOLLOWER]: Refusing update from remote peer 70664d89ba3545e5bbcf62a84f8c49b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:06:28.917666 8457 raft_consensus.cc:1261] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379 [term 1 FOLLOWER]: Refusing update from remote peer 70664d89ba3545e5bbcf62a84f8c49b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:06:28.918630 8527 consensus_queue.cc:1035] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:06:28.919021 8532 consensus_queue.cc:1035] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:06:28.931216 8535 mvcc.cc:205] Tried to move back new op lower bound from 7298586988196339712 to 7298586987876425728. Current Snapshot: MvccSnapshot[applied={T|T < 7298586988196339712}]
I20260619 15:06:28.932477 8537 mvcc.cc:205] Tried to move back new op lower bound from 7298586988196339712 to 7298586987876425728. Current Snapshot: MvccSnapshot[applied={T|T < 7298586988196339712}]
I20260619 15:06:28.948516 8044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:06:28.948894 8044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:06:28.953042 8044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Sending DeleteTablet for 3 replicas of tablet 641fd7dddc4c4713a7fab9cda9ce1c6f
I20260619 15:06:28.954314 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 641fd7dddc4c4713a7fab9cda9ce1c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:28 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:41192
I20260619 15:06:28.954407 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 641fd7dddc4c4713a7fab9cda9ce1c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:28 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:60372
I20260619 15:06:28.955348 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 8106
I20260619 15:06:28.955776 8437 tablet_service.cc:1514] Processing DeleteTablet for tablet 641fd7dddc4c4713a7fab9cda9ce1c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:28 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:43564
I20260619 15:06:28.963869 8548 tablet_replica.cc:330] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379: stopping tablet replica
I20260619 15:06:28.966252 8547 tablet_replica.cc:330] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3: stopping tablet replica
I20260619 15:06:28.966481 8546 tablet_replica.cc:330] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9: stopping tablet replica
I20260619 15:06:28.966811 8547 raft_consensus.cc:2229] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:06:28.967337 8546 raft_consensus.cc:2229] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:06:28.967499 8547 raft_consensus.cc:2258] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:06:28.968139 8548 raft_consensus.cc:2229] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:06:28.968262 8546 raft_consensus.cc:2258] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:06:28.968704 8548 raft_consensus.cc:2258] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:06:28.970767 8546 ts_tablet_manager.cc:1905] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:06:28.972172 8548 ts_tablet_manager.cc:1905] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:06:28.976806 8547 ts_tablet_manager.cc:1905] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:06:28.980787 8548 ts_tablet_manager.cc:1918] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 19 15:06:28 dist-test-slave-zpbw krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:06:28.981055 8548 log.cc:1196] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/641fd7dddc4c4713a7fab9cda9ce1c6f
I20260619 15:06:28.981513 8546 ts_tablet_manager.cc:1918] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:06:28.981699 8546 log.cc:1196] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/641fd7dddc4c4713a7fab9cda9ce1c6f
I20260619 15:06:28.982125 8548 ts_tablet_manager.cc:1939] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 961f6fae5e564d9a90ef4ce887500379: Deleting consensus metadata
I20260619 15:06:28.982321 8546 ts_tablet_manager.cc:1939] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 70664d89ba3545e5bbcf62a84f8c49b9: Deleting consensus metadata
I20260619 15:06:28.984372 8030 catalog_manager.cc:4841] TS 961f6fae5e564d9a90ef4ce887500379 (127.24.103.67:45535): tablet 641fd7dddc4c4713a7fab9cda9ce1c6f (table test-table [id=cf0999eacce5482f8b1b8a9ef7832d3f]) successfully deleted
I20260619 15:06:28.989363 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:28.956721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.24.103.65:34271)
0619 15:06:28.956932 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:28.956941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:28.969640 (+ 12699us) server_negotiation.cc:407] Connection header received
0619 15:06:28.969895 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:28.969900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:28.969986 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:28.970112 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:28.972192 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.973514 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.975628 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:28.976354 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:28.976450 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:28.982472 (+ 6022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:28.982491 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:28.982514 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:28.982584 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:28.986109 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:28.987031 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:28.987039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:28.987041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:28.987188 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:28.987603 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:28.987608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:28.987612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:28.987989 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:28.988324 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:28.988682 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 15:06:28.988926 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 15:06:28.990068 8229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:28.990340 8547 ts_tablet_manager.cc:1918] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:06:28.990530 8547 log.cc:1196] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/641fd7dddc4c4713a7fab9cda9ce1c6f
I20260619 15:06:28.991597 8547 ts_tablet_manager.cc:1939] T 641fd7dddc4c4713a7fab9cda9ce1c6f P 403c75060d464222b2a712e98303e3d3: Deleting consensus metadata
I20260619 15:06:28.993541 8030 catalog_manager.cc:4841] TS 403c75060d464222b2a712e98303e3d3 (127.24.103.66:43261): tablet 641fd7dddc4c4713a7fab9cda9ce1c6f (table test-table [id=cf0999eacce5482f8b1b8a9ef7832d3f]) successfully deleted
I20260619 15:06:29.101779 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 8106
W20260619 15:06:29.120105 8029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.103.65:34271 (error 108)
I20260619 15:06:29.120666 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 8240
I20260619 15:06:29.122258 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:29.121392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:41214)
0619 15:06:29.122000 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:29.122101 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":351,"threads_started":1}
W20260619 15:06:29.122578 8029 catalog_manager.cc:4584] TS 70664d89ba3545e5bbcf62a84f8c49b9 (127.24.103.65:34271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 641fd7dddc4c4713a7fab9cda9ce1c6f: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Jun 19 15:06:29 dist-test-slave-zpbw krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:06:29.137125 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:29.121539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.24.103.66:43261)
0619 15:06:29.121726 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:29.121731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:29.121751 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:06:29.122054 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:29.122060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:29.122138 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:29.122253 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:29.123804 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:29.124567 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:29.126104 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:29.126501 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:29.126643 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:29.131234 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:29.131250 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:29.131266 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:29.131325 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:29.134026 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:29.134767 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:29.134829 (+ 62us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:29.134842 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:29.135003 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:29.135429 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:29.135436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:29.135440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:29.135882 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:29.136205 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:29.136533 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 15:06:29.136757 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 15:06:29.137797 8363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:29.176376 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:29.175894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:41216)
0619 15:06:29.176106 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:29.176220 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 15:06:29.177531 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:29.177144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:41232)
0619 15:06:29.177315 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:29.177384 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 15:06:29.228243 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 8240
I20260619 15:06:29.237903 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:29.237458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:41240)
0619 15:06:29.237666 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:29.237753 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 15:06:29.239105 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:29.238712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:41246)
0619 15:06:29.238894 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:29.238969 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 15:06:29.245406 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 8374
Jun 19 15:06:29 dist-test-slave-zpbw krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:06:29.261361 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:29.246209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.24.103.67:45535)
0619 15:06:29.246418 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:29.246423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:29.246443 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:06:29.246526 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:29.246529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:29.246594 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:29.246680 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:29.248043 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:29.248800 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:29.250185 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:29.250582 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:29.250682 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:29.255583 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:29.255601 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:29.255617 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:29.255663 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:29.258601 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:29.259401 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:29.259408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:29.259410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:29.259555 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:29.259942 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:29.259946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:29.259948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:29.260345 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:29.260665 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:29.260869 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 15:06:29.261042 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 15:06:29.262049 8497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:29.319991 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:29.319465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:41262)
0619 15:06:29.319738 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:29.319843 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 15:06:29.321136 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:29.320714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:41266)
0619 15:06:29.320918 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:29.321001 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 15:06:29.351543 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 8374
I20260619 15:06:29.368352 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 8013
I20260619 15:06:29.369539 8075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:29.480082 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 8013
I20260619 15:06:49.501592 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34375
--webserver_interface=127.24.103.126
--webserver_port=41319
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:34375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:06:49.846525 8556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:49.846856 8556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:49.847062 8556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:49.854624 8556 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 15:06:49.854733 8556 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 15:06:49.854852 8556 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 15:06:49.854947 8556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:06:49.855073 8556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:49.855201 8556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:49.855278 8556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:06:49.855360 8556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:06:49.866021 8556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:34375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=41319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:49.867489 8556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:49.868868 8556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:49.892467 8565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:49.892544 8563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:49.892500 8562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:49.893412 8556 server_base.cc:1034] running on GCE node
I20260619 15:06:49.894273 8556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:49.896059 8556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:49.897284 8556 hybrid_clock.cc:648] HybridClock initialized: now 1781881609897221 us; error 32 us; skew 500 ppm
I20260619 15:06:49.897717 8556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:49 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881609, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:49.905208 8556 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:06:49.907061 8556 webserver.cc:460] Webserver started at http://127.24.103.126:41319/ using document root <none> and password file <none>
I20260619 15:06:49.907912 8556 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:49.908061 8556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:49.914649 8556 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 15:06:49.917900 8572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:49.919585 8556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:06:49.919781 8556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "e0f338b8b9c442269a35a15e86599a23"
format_stamp: "Formatted at 2026-06-19 15:06:26 on dist-test-slave-zpbw"
server_key: "ee169b68ebb9938aa69b5d7c5ab04da5"
server_key_iv: "1d74969d1a430f3ac1602eeec813fb93"
server_key_version: "encryptionkey@0"
I20260619 15:06:49.920907 8556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:49.942312 8556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:49.947062 8556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:49.947399 8556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:49.967808 8556 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:34375
I20260619 15:06:49.967830 8623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:34375 every 8 connection(s)
I20260619 15:06:49.969491 8556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:06:49.976858 8624 sys_catalog.cc:263] Verifying existing consensus state
I20260619 15:06:49.977378 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 8556
I20260619 15:06:49.978272 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:34271
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:34375
--webserver_port=36543
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:06:49.980623 8624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Bootstrap starting.
I20260619 15:06:50.001739 8624 log.cc:824] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:50.014714 8624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:06:50.015414 8624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Bootstrap complete.
I20260619 15:06:50.025089 8624 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } }
I20260619 15:06:50.026172 8624 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f338b8b9c442269a35a15e86599a23, State: Initialized, Role: FOLLOWER
I20260619 15:06:50.026957 8624 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } }
I20260619 15:06:50.027241 8624 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:06:50.027381 8624 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:06:50.027563 8624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 1 FOLLOWER]: Advancing to term 2
I20260619 15:06:50.031914 8624 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } }
I20260619 15:06:50.032542 8624 leader_election.cc:302] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0f338b8b9c442269a35a15e86599a23; no voters:
I20260619 15:06:50.033259 8624 leader_election.cc:288] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 15:06:50.033432 8628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 15:06:50.034858 8628 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 2 LEADER]: Becoming Leader. State: Replica: e0f338b8b9c442269a35a15e86599a23, State: Running, Role: LEADER
I20260619 15:06:50.035542 8628 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } }
I20260619 15:06:50.035897 8624 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:06:50.038129 8629 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e0f338b8b9c442269a35a15e86599a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } } }
I20260619 15:06:50.038513 8629 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: This master's current role is: LEADER
I20260619 15:06:50.042114 8630 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f338b8b9c442269a35a15e86599a23. Latest consensus state: current_term: 2 leader_uuid: "e0f338b8b9c442269a35a15e86599a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } } }
I20260619 15:06:50.042470 8630 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: This master's current role is: LEADER
I20260619 15:06:50.044426 8633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:06:50.055313 8633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:06:50.057814 8633 catalog_manager.cc:1259] Loaded cluster ID: 47a08e463fde4fa683ded23334679d38
I20260619 15:06:50.058084 8633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:06:50.063824 8633 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:06:50.066416 8633 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Loaded TSK: 0
I20260619 15:06:50.086668 8645 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 15:06:50.099944 8633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Generated new TSK 1
I20260619 15:06:50.100395 8633 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 15:06:50.367969 8626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:50.368330 8626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:50.368508 8626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:50.376255 8626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:50.376425 8626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:50.376608 8626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:06:50.386897 8626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:34271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=36543
--tserver_master_addrs=127.24.103.126:34375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:50.388298 8626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:50.389712 8626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:50.404796 8652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:50.405205 8654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:50.405447 8651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:50.406392 8626 server_base.cc:1034] running on GCE node
I20260619 15:06:50.407248 8626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:50.408353 8626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:50.409605 8626 hybrid_clock.cc:648] HybridClock initialized: now 1781881610409543 us; error 52 us; skew 500 ppm
I20260619 15:06:50.409957 8626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:50 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881610, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:50.413512 8626 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:06:50.415223 8626 webserver.cc:460] Webserver started at http://127.24.103.65:36543/ using document root <none> and password file <none>
I20260619 15:06:50.415951 8626 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:50.416106 8626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:50.422068 8626 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 15:06:50.425276 8661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:50.426892 8626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 15:06:50.427079 8626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "70664d89ba3545e5bbcf62a84f8c49b9"
format_stamp: "Formatted at 2026-06-19 15:06:27 on dist-test-slave-zpbw"
server_key: "86093e31dd646081e00156c208f3afbe"
server_key_iv: "48b4b49e6162dfd9c6fd5b6363022554"
server_key_version: "encryptionkey@0"
I20260619 15:06:50.428140 8626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:50.462437 8626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:50.466933 8626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:50.467247 8626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:50.468355 8626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:50.469767 8626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:50.469895 8626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:50.470033 8626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:50.470125 8626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:50.505295 8626 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:34271
I20260619 15:06:50.505347 8773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:34271 every 8 connection(s)
I20260619 15:06:50.507033 8626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:06:50.515640 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 8626
I20260619 15:06:50.516451 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:43261
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:34375
--webserver_port=38551
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 15:06:50 dist-test-slave-zpbw krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881610, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:50.534950 8776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:50.510050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:52351 (local address 127.24.103.126:34375)
0619 15:06:50.510704 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:50.510727 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:50.511404 (+ 677us) server_negotiation.cc:407] Connection header received
0619 15:06:50.513526 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:50.513539 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:50.513888 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:50.514214 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:50.516268 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:50.517626 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:50.520322 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:50.521011 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:50.521168 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:50.526355 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:50.526366 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:50.526384 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:50.526454 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:50.529820 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:50.530927 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:50.530935 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:50.530941 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:50.531114 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:50.531674 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:50.531678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:50.531680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:50.532283 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:50.532455 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:50.533484 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0619 15:06:50.533842 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":253,"threads_started":1}
I20260619 15:06:50.535401 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:50.510299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:52351)
0619 15:06:50.511210 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:50.511256 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:50.512998 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:50.514379 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:50.514387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:50.515163 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:06:50.516029 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:50.516056 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:50.517847 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:50.517855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:50.520133 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:50.520154 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:50.521292 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:50.521302 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:50.521659 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:50.522478 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:06:50.522514 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:06:50.526000 (+ 3486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:06:50.530016 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:50.530029 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:50.530050 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:50.530753 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:50.531244 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:50.531252 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:50.531258 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:50.531546 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:50.532457 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:06:50.532468 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:06:50.532957 (+ 489us) client_negotiation.cc:769] Sending connection context
0619 15:06:50.533273 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 15:06:50.533919 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":391,"threads_started":1}
I20260619 15:06:50.538884 8774 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34375
I20260619 15:06:50.539239 8774 heartbeater.cc:464] Registering TS with master...
I20260619 15:06:50.540028 8774 heartbeater.cc:511] Master 127.24.103.126:34375 requested a full tablet report, sending...
I20260619 15:06:50.542884 8589 ts_manager.cc:194] Registered new tserver with Master: 70664d89ba3545e5bbcf62a84f8c49b9 (127.24.103.65:34271)
I20260619 15:06:50.545532 8589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:52351
W20260619 15:06:50.880972 8778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:50.881386 8778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:50.881542 8778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:50.889727 8778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:50.889878 8778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:50.890110 8778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:06:50.901863 8778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:43261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=38551
--tserver_master_addrs=127.24.103.126:34375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:50.903294 8778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:50.904619 8778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:50.918501 8783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:50.918430 8784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:50.918467 8786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:50.919286 8778 server_base.cc:1034] running on GCE node
I20260619 15:06:50.920002 8778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:50.921000 8778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:50.922279 8778 hybrid_clock.cc:648] HybridClock initialized: now 1781881610922238 us; error 58 us; skew 500 ppm
I20260619 15:06:50.922652 8778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:50 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881610, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:50.926095 8778 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:06:50.927687 8778 webserver.cc:460] Webserver started at http://127.24.103.66:38551/ using document root <none> and password file <none>
I20260619 15:06:50.928371 8778 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:50.928503 8778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:50.934273 8778 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 15:06:50.937338 8793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:50.938935 8778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 15:06:50.939088 8778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "403c75060d464222b2a712e98303e3d3"
format_stamp: "Formatted at 2026-06-19 15:06:27 on dist-test-slave-zpbw"
server_key: "ab8c58584ba836841ee0c9df5444b7f4"
server_key_iv: "70bc673bfa569ed8ebfdd17872d915a3"
server_key_version: "encryptionkey@0"
I20260619 15:06:50.940050 8778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:50.957736 8778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:50.962100 8778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:50.962332 8778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:50.963335 8778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:50.964633 8778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:50.964727 8778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:50.964798 8778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:50.964855 8778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:51.000553 8778 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:43261
I20260619 15:06:51.000638 8905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:43261 every 8 connection(s)
I20260619 15:06:51.002064 8778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:06:51.006889 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 8778
I20260619 15:06:51.007684 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:45535
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:34375
--webserver_port=39001
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 15:06:51 dist-test-slave-zpbw krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881610, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:51.026503 8776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.004850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:48103 (local address 127.24.103.126:34375)
0619 15:06:51.005027 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:51.005031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:51.006183 (+ 1152us) server_negotiation.cc:407] Connection header received
0619 15:06:51.009134 (+ 2951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:51.009138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:51.009250 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:51.009342 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:51.011713 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.012450 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.015196 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.015642 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.015761 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.020027 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:51.020033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:51.020037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:51.020074 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:51.022731 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:51.023851 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:51.023855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:51.023858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:51.023983 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:51.024575 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:51.024578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:51.024580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:51.024872 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:51.024998 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:51.025976 (+ 978us) server_negotiation.cc:299] Negotiation successful
0619 15:06:51.026242 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 15:06:51.028292 8908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.005290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:48103)
0619 15:06:51.006014 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:51.006057 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:51.008766 (+ 2709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:51.009651 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:51.009667 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:51.010538 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:06:51.011456 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.011487 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.012638 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.012646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.014969 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.015010 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.015910 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.015920 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.016184 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.016967 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:06:51.016990 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:06:51.019720 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:06:51.022974 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:51.022988 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:51.023011 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:51.023713 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:51.024095 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:51.024103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:51.024110 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:51.024392 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:51.025016 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:06:51.025027 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:06:51.025538 (+ 511us) client_negotiation.cc:769] Sending connection context
0619 15:06:51.025892 (+ 354us) client_negotiation.cc:240] Negotiation successful
0619 15:06:51.026568 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":299,"threads_started":1}
I20260619 15:06:51.030355 8906 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34375
I20260619 15:06:51.030741 8906 heartbeater.cc:464] Registering TS with master...
I20260619 15:06:51.031867 8906 heartbeater.cc:511] Master 127.24.103.126:34375 requested a full tablet report, sending...
I20260619 15:06:51.033779 8589 ts_manager.cc:194] Registered new tserver with Master: 403c75060d464222b2a712e98303e3d3 (127.24.103.66:43261)
I20260619 15:06:51.034638 8589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:48103
W20260619 15:06:51.355963 8909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:06:51.356328 8909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:06:51.356513 8909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:06:51.364218 8909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:06:51.364393 8909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:06:51.364585 8909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:06:51.374706 8909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:45535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=39001
--tserver_master_addrs=127.24.103.126:34375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:06:51.376183 8909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:06:51.377547 8909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:06:51.391194 8917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:51.391244 8915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:06:51.391294 8914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:06:51.392104 8909 server_base.cc:1034] running on GCE node
I20260619 15:06:51.392751 8909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:06:51.393765 8909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:06:51.395005 8909 hybrid_clock.cc:648] HybridClock initialized: now 1781881611394940 us; error 54 us; skew 500 ppm
I20260619 15:06:51.395349 8909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:06:51 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881611, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:06:51.398497 8909 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:06:51.400197 8909 webserver.cc:460] Webserver started at http://127.24.103.67:39001/ using document root <none> and password file <none>
I20260619 15:06:51.400940 8909 fs_manager.cc:359] Metadata directory not provided
I20260619 15:06:51.401109 8909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:06:51.407096 8909 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 15:06:51.410113 8924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:51.411664 8909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:06:51.411863 8909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "961f6fae5e564d9a90ef4ce887500379"
format_stamp: "Formatted at 2026-06-19 15:06:28 on dist-test-slave-zpbw"
server_key: "19673095930822a7026bf497f3b6df18"
server_key_iv: "5bb97b51dbd10eb9d45466db67a1a019"
server_key_version: "encryptionkey@0"
I20260619 15:06:51.412976 8909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:06:51.435159 8909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:06:51.439848 8909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:06:51.440163 8909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:06:51.441601 8909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:06:51.443109 8909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:06:51.443243 8909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:51.443373 8909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:06:51.443463 8909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:06:51.478367 8909 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:45535
I20260619 15:06:51.478461 9036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:45535 every 8 connection(s)
I20260619 15:06:51.479959 8909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:06:51.487798 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 8909
Jun 19 15:06:51 dist-test-slave-zpbw krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881611, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:06:51.502803 8776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.482608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:45815 (local address 127.24.103.126:34375)
0619 15:06:51.482850 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:51.482856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:51.484085 (+ 1229us) server_negotiation.cc:407] Connection header received
0619 15:06:51.485944 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:51.485949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:51.486038 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:51.486132 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:51.488085 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.489233 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.491040 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.491788 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.491946 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.496105 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:51.496112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:51.496118 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:51.496163 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:51.499425 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:51.500263 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:51.500273 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:51.500277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:51.500424 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:51.500916 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:51.500926 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:51.500928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:51.501219 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:51.501378 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:51.502151 (+ 773us) server_negotiation.cc:299] Negotiation successful
0619 15:06:51.502383 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 15:06:51.504101 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.483180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:45815)
0619 15:06:51.483919 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:51.483958 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:51.485638 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:51.486361 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:51.486370 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:51.487017 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:06:51.487825 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.487854 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.489515 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.489520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.490895 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.490907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.492023 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.492028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.492405 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.493152 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:06:51.493188 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:06:51.495812 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:06:51.499592 (+ 3780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:51.499600 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:51.499621 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:51.500116 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:51.500556 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:06:51.500561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:06:51.500565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:06:51.500777 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:06:51.501355 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:06:51.501365 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:06:51.501757 (+ 392us) client_negotiation.cc:769] Sending connection context
0619 15:06:51.502049 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 15:06:51.502633 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":288,"threads_started":1}
I20260619 15:06:51.504760 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.488892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.24.103.126:34375)
0619 15:06:51.489443 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:51.489448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:51.489749 (+ 301us) server_negotiation.cc:407] Connection header received
0619 15:06:51.489967 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:51.489971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:51.490031 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:51.490145 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:51.492211 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.493260 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.494704 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.495254 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.495368 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.498390 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:51.498397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:51.498402 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:51.498463 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:51.501334 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:51.502144 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:51.502149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:51.502153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:51.502274 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:51.502696 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:51.502702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:51.502706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:51.503049 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:51.503385 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:51.503679 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 15:06:51.503901 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":266,"threads_started":1}
I20260619 15:06:51.506443 9037 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34375
I20260619 15:06:51.506808 9037 heartbeater.cc:464] Registering TS with master...
I20260619 15:06:51.507714 9037 heartbeater.cc:511] Master 127.24.103.126:34375 requested a full tablet report, sending...
I20260619 15:06:51.509430 8588 ts_manager.cc:194] Registered new tserver with Master: 961f6fae5e564d9a90ef4ce887500379 (127.24.103.67:45535)
I20260619 15:06:51.510372 8588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:45815
I20260619 15:06:51.520252 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:06:51.527405 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.521254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46494 (local address 127.24.103.126:34375)
0619 15:06:51.521457 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:51.521462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:51.521973 (+ 511us) server_negotiation.cc:407] Connection header received
0619 15:06:51.522177 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:51.522181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:51.522330 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:51.522471 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:06:51.522966 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.523684 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.525317 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.525825 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.525931 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.526497 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:06:51.526739 (+ 242us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 15:06:51.527172 (+ 433us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46494: token expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 15:06:51.527516 9040 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46494: token expired
W20260619 15:06:51.528165 8508 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.103.126:34375)
I20260619 15:06:51.542292 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.528704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.24.103.126:34375)
0619 15:06:51.528916 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:51.528924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:51.528943 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:06:51.529132 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:51.529138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:51.529212 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:51.529322 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:51.530844 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.531619 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.533178 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.533592 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.533740 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.536663 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:51.536669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:51.536709 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:51.536758 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:51.539372 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:51.540188 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:51.540195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:51.540199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:51.540351 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:51.540746 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:51.540752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:51.540755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:51.541106 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:51.541379 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:51.541694 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 15:06:51.541945 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 15:06:51.549594 8774 heartbeater.cc:503] Master 127.24.103.126:34375 was elected leader, sending a full tablet report...
I20260619 15:06:51.545924 8588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:06:51.554275 8588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:06:51.588868 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.567015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.24.103.67:45535)
0619 15:06:51.578504 (+ 11489us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:51.578517 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:51.578565 (+ 48us) server_negotiation.cc:407] Connection header received
0619 15:06:51.578707 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:51.578720 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:51.579094 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:51.579363 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:51.580407 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.582561 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.586340 (+ 3779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.587623 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.587795 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.588059 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:51.588330 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 15:06:51.588500 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11269,"thread_start_us":11149,"threads_started":1}
I20260619 15:06:51.588954 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.566801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:45535 (local address 127.0.0.1:59890)
0619 15:06:51.572573 (+ 5772us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:51.572633 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:51.572902 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:51.579567 (+ 6665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:51.579573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:51.579605 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:51.580171 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.580184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.582880 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.582888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.586179 (+ 3291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.586191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.587861 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.587867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.588063 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.588134 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 15:06:51.588286 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 15:06:51.588501 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5381,"mutex_wait_us":87,"thread_start_us":5181,"threads_started":1}
I20260619 15:06:51.590117 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.568074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.24.103.65:34271)
0619 15:06:51.578118 (+ 10044us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:51.578132 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:51.579294 (+ 1162us) server_negotiation.cc:407] Connection header received
0619 15:06:51.579403 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:51.579410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:51.579656 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:51.579860 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:51.580795 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.582269 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.586950 (+ 4681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.588458 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.588625 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.588865 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:51.589624 (+ 759us) server_negotiation.cc:299] Negotiation successful
0619 15:06:51.589802 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9762,"thread_start_us":9543,"threads_started":1}
I20260619 15:06:51.590128 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.577314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:43261 (local address 127.0.0.1:47664)
0619 15:06:51.580255 (+ 2941us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:51.580278 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:51.580430 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:51.581433 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:51.581437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:51.581466 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:51.581853 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.581864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.583840 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.583846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.586969 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.586983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.589016 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.589025 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.589328 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.589445 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 15:06:51.589540 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 15:06:51.589726 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2723,"mutex_wait_us":791,"thread_start_us":2534,"threads_started":1}
I20260619 15:06:51.590759 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.567630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:46708)
0619 15:06:51.578995 (+ 11365us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:51.579018 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:51.579156 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:51.580136 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:51.580146 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:51.580171 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:51.580588 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.580603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.582422 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.583283 (+ 861us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.586753 (+ 3470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.586767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.589036 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.589041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.589333 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.589450 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 15:06:51.589540 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 15:06:51.589726 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11215,"mutex_wait_us":1218,"thread_start_us":11011,"threads_started":1}
I20260619 15:06:51.590916 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.573500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.24.103.66:43261)
0619 15:06:51.580959 (+ 7459us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:51.580969 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:51.580997 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:06:51.581087 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:51.581092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:51.581242 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:51.581437 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:51.582042 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.583656 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.587143 (+ 3487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.588551 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.588673 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.588831 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:51.590370 (+ 1539us) server_negotiation.cc:299] Negotiation successful
0619 15:06:51.590587 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7301,"thread_start_us":7180,"threads_started":1}
I20260619 15:06:51.593844 8841 tablet_service.cc:1467] Processing CreateTablet for tablet e30b237a624a4a3cb909dfc2eaba3056 (DEFAULT_TABLE table=test-table [id=7377bc83b0934bc3b1afe7da0c8b6263]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:06:51.593641 8972 tablet_service.cc:1467] Processing CreateTablet for tablet e30b237a624a4a3cb909dfc2eaba3056 (DEFAULT_TABLE table=test-table [id=7377bc83b0934bc3b1afe7da0c8b6263]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:06:51.594602 8709 tablet_service.cc:1467] Processing CreateTablet for tablet e30b237a624a4a3cb909dfc2eaba3056 (DEFAULT_TABLE table=test-table [id=7377bc83b0934bc3b1afe7da0c8b6263]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:06:51.595225 8841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30b237a624a4a3cb909dfc2eaba3056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:51.595237 8972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30b237a624a4a3cb909dfc2eaba3056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:51.596045 8709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30b237a624a4a3cb909dfc2eaba3056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:06:51.607743 9053 tablet_bootstrap.cc:492] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3: Bootstrap starting.
I20260619 15:06:51.611073 9055 tablet_bootstrap.cc:492] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379: Bootstrap starting.
I20260619 15:06:51.612656 9053 tablet_bootstrap.cc:654] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:51.612743 9054 tablet_bootstrap.cc:492] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9: Bootstrap starting.
I20260619 15:06:51.614217 9053 log.cc:824] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:51.615356 9055 tablet_bootstrap.cc:654] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:51.616129 9054 tablet_bootstrap.cc:654] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9: Neither blocks nor log segments found. Creating new log.
I20260619 15:06:51.616508 9055 log.cc:824] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:51.617458 9054 log.cc:824] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9: Log is configured to *not* fsync() on all Append() calls
I20260619 15:06:51.618003 9053 tablet_bootstrap.cc:492] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3: No bootstrap required, opened a new log
I20260619 15:06:51.618414 9053 ts_tablet_manager.cc:1397] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260619 15:06:51.619684 9055 tablet_bootstrap.cc:492] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379: No bootstrap required, opened a new log
I20260619 15:06:51.619967 9055 ts_tablet_manager.cc:1397] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 15:06:51.626009 9054 tablet_bootstrap.cc:492] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9: No bootstrap required, opened a new log
I20260619 15:06:51.626363 9054 ts_tablet_manager.cc:1397] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260619 15:06:51.628011 9055 raft_consensus.cc:348] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } } peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } }
I20260619 15:06:51.628427 9055 raft_consensus.cc:374] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:51.628579 9055 raft_consensus.cc:729] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 961f6fae5e564d9a90ef4ce887500379, State: Initialized, Role: FOLLOWER
I20260619 15:06:51.629398 9055 consensus_queue.cc:260] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } } peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } }
I20260619 15:06:51.630816 9053 raft_consensus.cc:348] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } } peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } }
I20260619 15:06:51.631326 9053 raft_consensus.cc:374] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:51.631633 9037 heartbeater.cc:503] Master 127.24.103.126:34375 was elected leader, sending a full tablet report...
I20260619 15:06:51.632376 9055 ts_tablet_manager.cc:1428] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260619 15:06:51.633042 9053 raft_consensus.cc:729] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403c75060d464222b2a712e98303e3d3, State: Initialized, Role: FOLLOWER
I20260619 15:06:51.633860 9053 consensus_queue.cc:260] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } } peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } }
I20260619 15:06:51.635767 8906 heartbeater.cc:503] Master 127.24.103.126:34375 was elected leader, sending a full tablet report...
I20260619 15:06:51.637988 9053 ts_tablet_manager.cc:1428] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3: Time spent starting tablet: real 0.019s user 0.008s sys 0.007s
I20260619 15:06:51.637848 9054 raft_consensus.cc:348] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } } peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } }
I20260619 15:06:51.638386 9054 raft_consensus.cc:374] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:06:51.638499 9054 raft_consensus.cc:729] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70664d89ba3545e5bbcf62a84f8c49b9, State: Initialized, Role: FOLLOWER
I20260619 15:06:51.639297 9054 consensus_queue.cc:260] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } } peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } }
I20260619 15:06:51.641049 9054 ts_tablet_manager.cc:1428] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260619 15:06:51.731721 9038 tablet.cc:2369] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:06:51.754279 8907 tablet.cc:2369] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:06:51.759476 8775 tablet.cc:2369] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:06:51.766995 9061 raft_consensus.cc:482] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:06:51.767324 9061 raft_consensus.cc:504] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } } peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } }
I20260619 15:06:51.769066 9061 leader_election.cc:288] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 961f6fae5e564d9a90ef4ce887500379 (127.24.103.67:45535), 403c75060d464222b2a712e98303e3d3 (127.24.103.66:43261)
I20260619 15:06:51.777475 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.769545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:44753 (local address 127.24.103.67:45535)
0619 15:06:51.769749 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:51.769754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:51.770408 (+ 654us) server_negotiation.cc:407] Connection header received
0619 15:06:51.770660 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:51.770670 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:51.770798 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:51.770883 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:51.771801 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.772848 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.775411 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.776587 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.776764 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.776843 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:51.777107 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 15:06:51.777252 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:06:51.777662 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.769550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:45535 (local address 127.24.103.65:44753)
0619 15:06:51.770293 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:51.770316 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:51.770464 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:51.771170 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:51.771174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:51.771216 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:51.771649 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.771664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.772985 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.772988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.775243 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.775259 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.776784 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.776788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.776919 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.776944 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 15:06:51.777064 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 15:06:51.777223 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":1,"thread_start_us":421,"threads_started":1}
I20260619 15:06:51.777913 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.770651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:51111 (local address 127.24.103.66:43261)
0619 15:06:51.770870 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:51.770877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:51.771539 (+ 662us) server_negotiation.cc:407] Connection header received
0619 15:06:51.771757 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:51.771763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:51.771836 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:51.771926 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:06:51.772514 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.773357 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.775701 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.776934 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.777092 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.777158 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:51.777502 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 15:06:51.777655 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:06:51.778024 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.770457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:43261 (local address 127.24.103.65:51111)
0619 15:06:51.771415 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:51.771439 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 15:06:51.771600 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:06:51.771963 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:06:51.771967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:06:51.771992 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:06:51.772342 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.772351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.773507 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.773512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.775550 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:06:51.775560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.777151 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.777157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:06:51.777306 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.777346 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 15:06:51.777454 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 15:06:51.777572 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":10,"thread_start_us":587,"threads_started":1}
I20260619 15:06:51.778880 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30b237a624a4a3cb909dfc2eaba3056" candidate_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "961f6fae5e564d9a90ef4ce887500379" is_pre_election: true
I20260619 15:06:51.779050 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30b237a624a4a3cb909dfc2eaba3056" candidate_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403c75060d464222b2a712e98303e3d3" is_pre_election: true
I20260619 15:06:51.779444 8992 raft_consensus.cc:2454] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70664d89ba3545e5bbcf62a84f8c49b9 in term 0.
I20260619 15:06:51.779558 8861 raft_consensus.cc:2454] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70664d89ba3545e5bbcf62a84f8c49b9 in term 0.
I20260619 15:06:51.780115 8663 leader_election.cc:302] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70664d89ba3545e5bbcf62a84f8c49b9, 961f6fae5e564d9a90ef4ce887500379; no voters:
I20260619 15:06:51.780534 9061 raft_consensus.cc:2781] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:06:51.780690 9061 raft_consensus.cc:482] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:06:51.780802 9061 raft_consensus.cc:3037] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:51.783085 9061 raft_consensus.cc:504] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } } peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } }
I20260619 15:06:51.783705 9061 leader_election.cc:288] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [CANDIDATE]: Term 1 election: Requested vote from peers 961f6fae5e564d9a90ef4ce887500379 (127.24.103.67:45535), 403c75060d464222b2a712e98303e3d3 (127.24.103.66:43261)
I20260619 15:06:51.784152 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30b237a624a4a3cb909dfc2eaba3056" candidate_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "961f6fae5e564d9a90ef4ce887500379"
I20260619 15:06:51.784226 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30b237a624a4a3cb909dfc2eaba3056" candidate_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403c75060d464222b2a712e98303e3d3"
I20260619 15:06:51.784368 8992 raft_consensus.cc:3037] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:51.784471 8861 raft_consensus.cc:3037] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:06:51.786667 8992 raft_consensus.cc:2454] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70664d89ba3545e5bbcf62a84f8c49b9 in term 1.
I20260619 15:06:51.786712 8861 raft_consensus.cc:2454] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70664d89ba3545e5bbcf62a84f8c49b9 in term 1.
I20260619 15:06:51.787132 8663 leader_election.cc:302] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403c75060d464222b2a712e98303e3d3, 70664d89ba3545e5bbcf62a84f8c49b9; no voters:
I20260619 15:06:51.787405 9061 raft_consensus.cc:2781] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:06:51.787860 9061 raft_consensus.cc:686] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 1 LEADER]: Becoming Leader. State: Replica: 70664d89ba3545e5bbcf62a84f8c49b9, State: Running, Role: LEADER
I20260619 15:06:51.788353 9061 consensus_queue.cc:237] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } } peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } }
I20260619 15:06:51.792946 8588 catalog_manager.cc:5495] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70664d89ba3545e5bbcf62a84f8c49b9 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70664d89ba3545e5bbcf62a84f8c49b9" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34271 } health_report { overall_health: HEALTHY } } }
I20260619 15:06:51.830344 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.824563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.24.103.65:34271)
0619 15:06:51.824823 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:51.824830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:51.824958 (+ 128us) server_negotiation.cc:407] Connection header received
0619 15:06:51.825233 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:51.825237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:51.825338 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:51.825515 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:06:51.825997 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.826727 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.828140 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.828525 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.828620 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.829195 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:06:51.829736 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:06:51.829864 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:51.829967 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 15:06:51.830037 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 15:06:51.839228 8861 raft_consensus.cc:1261] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3 [term 1 FOLLOWER]: Refusing update from remote peer 70664d89ba3545e5bbcf62a84f8c49b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:06:51.839257 8992 raft_consensus.cc:1261] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379 [term 1 FOLLOWER]: Refusing update from remote peer 70664d89ba3545e5bbcf62a84f8c49b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:06:51.840251 9064 consensus_queue.cc:1035] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "403c75060d464222b2a712e98303e3d3" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:06:51.840658 9061 consensus_queue.cc:1035] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "961f6fae5e564d9a90ef4ce887500379" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:06:51.871009 8588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:06:51.871301 8588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:06:51.875586 8588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Sending DeleteTablet for 3 replicas of tablet e30b237a624a4a3cb909dfc2eaba3056
I20260619 15:06:51.877117 8841 tablet_service.cc:1514] Processing DeleteTablet for tablet e30b237a624a4a3cb909dfc2eaba3056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:51 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:47664
I20260619 15:06:51.877167 8972 tablet_service.cc:1514] Processing DeleteTablet for tablet e30b237a624a4a3cb909dfc2eaba3056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:51 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:59890
I20260619 15:06:51.877370 8709 tablet_service.cc:1514] Processing DeleteTablet for tablet e30b237a624a4a3cb909dfc2eaba3056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:51 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:46708
I20260619 15:06:51.877861 8971 tablet_service.cc:1514] Processing DeleteTablet for tablet e30b237a624a4a3cb909dfc2eaba3056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:06:51 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:59890
I20260619 15:06:51.878547 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 8626
I20260619 15:06:51.888511 9076 tablet_replica.cc:330] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379: stopping tablet replica
I20260619 15:06:51.889209 9076 raft_consensus.cc:2229] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:06:51.889719 9078 tablet_replica.cc:330] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9: stopping tablet replica
I20260619 15:06:51.890375 9076 raft_consensus.cc:2258] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:06:51.890479 9078 raft_consensus.cc:2229] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:06:51.891445 9078 raft_consensus.cc:2258] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:06:51.892107 9077 tablet_replica.cc:330] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3: stopping tablet replica
I20260619 15:06:51.892489 9077 raft_consensus.cc:2229] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:06:51.892925 9077 raft_consensus.cc:2258] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:06:51.892997 9076 ts_tablet_manager.cc:1905] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:06:51.895107 9078 ts_tablet_manager.cc:1905] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:06:51.895210 9077 ts_tablet_manager.cc:1905] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:06:51.902644 9077 ts_tablet_manager.cc:1918] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:06:51.903256 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:51.879803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46728 (local address 127.24.103.65:34271)
0619 15:06:51.880041 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:51.880051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:51.881049 (+ 998us) server_negotiation.cc:407] Connection header received
0619 15:06:51.881140 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:51.881148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:51.881312 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:51.881429 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:51.884139 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.885775 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.888432 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:51.889136 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:51.889294 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:51.893683 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:51.893705 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:51.893723 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:51.893787 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:51.898398 (+ 4611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:51.899570 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:51.899580 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:51.899583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:51.899802 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:51.900365 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:51.900370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:51.900373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:51.900869 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:51.901231 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:51.902531 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0619 15:06:51.902817 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 15:06:51.903743 9078 ts_tablet_manager.cc:1918] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:06:51.903935 9078 log.cc:1196] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/e30b237a624a4a3cb909dfc2eaba3056
I20260619 15:06:51.904125 8769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:51.908876 9077 log.cc:1196] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/e30b237a624a4a3cb909dfc2eaba3056
I20260619 15:06:51.909504 9077 ts_tablet_manager.cc:1939] T e30b237a624a4a3cb909dfc2eaba3056 P 403c75060d464222b2a712e98303e3d3: Deleting consensus metadata
I20260619 15:06:51.911085 9076 ts_tablet_manager.cc:1918] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:06:51.911262 9076 log.cc:1196] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/e30b237a624a4a3cb909dfc2eaba3056
I20260619 15:06:51.911638 8574 catalog_manager.cc:4841] TS 403c75060d464222b2a712e98303e3d3 (127.24.103.66:43261): tablet e30b237a624a4a3cb909dfc2eaba3056 (table test-table [id=7377bc83b0934bc3b1afe7da0c8b6263]) successfully deleted
I20260619 15:06:51.911932 9076 ts_tablet_manager.cc:1939] T e30b237a624a4a3cb909dfc2eaba3056 P 961f6fae5e564d9a90ef4ce887500379: Deleting consensus metadata
I20260619 15:06:51.913441 8574 catalog_manager.cc:4841] TS 961f6fae5e564d9a90ef4ce887500379 (127.24.103.67:45535): tablet e30b237a624a4a3cb909dfc2eaba3056 (table test-table [id=7377bc83b0934bc3b1afe7da0c8b6263]) successfully deleted
W20260619 15:06:51.914011 8574 catalog_manager.cc:4808] TS 961f6fae5e564d9a90ef4ce887500379 (127.24.103.67:45535): delete failed for tablet e30b237a624a4a3cb909dfc2eaba3056 because the tablet was not found. No further retry: Not found: Tablet not found: e30b237a624a4a3cb909dfc2eaba3056
I20260619 15:06:52.006852 9078 ts_tablet_manager.cc:1939] T e30b237a624a4a3cb909dfc2eaba3056 P 70664d89ba3545e5bbcf62a84f8c49b9: Deleting consensus metadata
I20260619 15:06:52.007316 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 8626
W20260619 15:06:52.025779 8573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.103.65:34271 (error 108)
I20260619 15:06:52.026022 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 8778
I20260619 15:06:52.027952 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:52.027517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:46744)
0619 15:06:52.027709 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:52.027817 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 15:06:52.028182 8573 catalog_manager.cc:4584] TS 70664d89ba3545e5bbcf62a84f8c49b9 (127.24.103.65:34271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e30b237a624a4a3cb909dfc2eaba3056: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
I20260619 15:06:52.045379 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:52.026940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.24.103.66:43261)
0619 15:06:52.027104 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:52.027108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:52.027152 (+ 44us) server_negotiation.cc:407] Connection header received
0619 15:06:52.027333 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:52.027338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:52.027400 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:52.027493 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:52.029165 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:52.029963 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:52.031598 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:52.032023 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:52.032150 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:52.039358 (+ 7208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:52.039377 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:52.039393 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:52.039443 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:52.042193 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:52.043069 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:52.043077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:52.043079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:52.043206 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:52.043549 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:52.043552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:52.043554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:52.044060 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:52.044386 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:52.044767 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 15:06:52.045020 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 15:06:52.046087 8901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:52.081756 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:52.081272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:46752)
0619 15:06:52.081502 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:52.081597 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 15:06:52.082814 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:52.082434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:46756)
0619 15:06:52.082575 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:52.082662 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 15:06:52.136547 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 8778
I20260619 15:06:52.143487 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:52.143032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:46760)
0619 15:06:52.143228 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:52.143315 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 15:06:52.144706 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:52.144306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:46762)
0619 15:06:52.144453 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:52.144593 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 15:06:52.154371 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 8909
I20260619 15:06:52.168054 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:52.155226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.24.103.67:45535)
0619 15:06:52.155412 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 15:06:52.155417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:06:52.155437 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:06:52.155559 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:06:52.155564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:06:52.155639 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:06:52.155723 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:06:52.157081 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:52.157770 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:52.159314 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:06:52.159700 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:06:52.159799 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:06:52.162289 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:06:52.162304 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:06:52.162320 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:06:52.162354 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:06:52.164991 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:52.165772 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:52.165779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:52.165782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:52.165968 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:06:52.166379 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:06:52.166382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:06:52.166384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:06:52.166745 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:06:52.167250 (+ 505us) server_negotiation.cc:1035] Waiting for connection context
0619 15:06:52.167519 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 15:06:52.167689 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:06:52.168751 9032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:52.225540 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:52.224911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:46776)
0619 15:06:52.225199 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:52.225318 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260619 15:06:52.226873 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:06:52.226335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34271 (local address 127.0.0.1:46780)
0619 15:06:52.226584 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 15:06:52.226690 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260619 15:06:52.258704 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 8909
I20260619 15:06:52.275315 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 8556
I20260619 15:06:52.276468 8619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:06:52.391150 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 8556
I20260619 15:07:12.413239 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34375
--webserver_interface=127.24.103.126
--webserver_port=41319
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:34375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:12.757655 9084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:12.758039 9084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:12.758143 9084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:12.765801 9084 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 15:07:12.765929 9084 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 15:07:12.766038 9084 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 15:07:12.766191 9084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:07:12.766307 9084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:12.766491 9084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:12.766608 9084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:07:12.766719 9084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:07:12.777493 9084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:34375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=41319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:12.778935 9084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:12.780357 9084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:12.793743 9089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:12.793743 9090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:12.793768 9092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:12.794143 9084 server_base.cc:1034] running on GCE node
I20260619 15:07:12.795449 9084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:12.797053 9084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:12.798319 9084 hybrid_clock.cc:648] HybridClock initialized: now 1781881632798235 us; error 66 us; skew 500 ppm
I20260619 15:07:12.798672 9084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:12 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881632, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:12.802156 9084 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:07:12.803732 9084 webserver.cc:460] Webserver started at http://127.24.103.126:41319/ using document root <none> and password file <none>
I20260619 15:07:12.804540 9084 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:12.804636 9084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:12.810936 9084 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 15:07:12.813941 9099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:12.815503 9084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:07:12.815718 9084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "e0f338b8b9c442269a35a15e86599a23"
format_stamp: "Formatted at 2026-06-19 15:06:26 on dist-test-slave-zpbw"
server_key: "ee169b68ebb9938aa69b5d7c5ab04da5"
server_key_iv: "1d74969d1a430f3ac1602eeec813fb93"
server_key_version: "encryptionkey@0"
I20260619 15:07:12.816771 9084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:12.836302 9084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:12.840767 9084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:12.841087 9084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:12.861465 9084 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:34375
I20260619 15:07:12.861483 9150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:34375 every 8 connection(s)
I20260619 15:07:12.863611 9084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:07:12.868953 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 9084
I20260619 15:07:12.869899 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:34271
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:34375
--webserver_port=36543
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:07:12.871657 9151 sys_catalog.cc:263] Verifying existing consensus state
I20260619 15:07:12.875195 9151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Bootstrap starting.
I20260619 15:07:12.900827 9151 log.cc:824] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:12.918391 9151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:07:12.919090 9151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Bootstrap complete.
I20260619 15:07:12.929147 9151 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } }
I20260619 15:07:12.930167 9151 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f338b8b9c442269a35a15e86599a23, State: Initialized, Role: FOLLOWER
I20260619 15:07:12.930938 9151 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } }
I20260619 15:07:12.931206 9151 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:07:12.931352 9151 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:07:12.931545 9151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 2 FOLLOWER]: Advancing to term 3
I20260619 15:07:12.935752 9151 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } }
I20260619 15:07:12.936326 9151 leader_election.cc:302] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0f338b8b9c442269a35a15e86599a23; no voters:
I20260619 15:07:12.937060 9151 leader_election.cc:288] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 15:07:12.937209 9155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 15:07:12.938510 9155 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [term 3 LEADER]: Becoming Leader. State: Replica: e0f338b8b9c442269a35a15e86599a23, State: Running, Role: LEADER
I20260619 15:07:12.939168 9155 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } }
I20260619 15:07:12.939550 9151 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:07:12.941864 9157 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f338b8b9c442269a35a15e86599a23. Latest consensus state: current_term: 3 leader_uuid: "e0f338b8b9c442269a35a15e86599a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } } }
I20260619 15:07:12.942356 9157 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:12.943070 9156 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e0f338b8b9c442269a35a15e86599a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f338b8b9c442269a35a15e86599a23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34375 } } }
I20260619 15:07:12.943506 9156 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:12.947762 9161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:07:12.965140 9161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:07:12.967325 9161 catalog_manager.cc:1259] Loaded cluster ID: 47a08e463fde4fa683ded23334679d38
I20260619 15:07:12.967564 9161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:07:12.972157 9161 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:07:12.974195 9161 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Loaded TSK: 0
I20260619 15:07:12.974511 9161 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0f338b8b9c442269a35a15e86599a23: Loaded TSK: 1
F20260619 15:07:12.976030 9161 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f986ec009ad google::LogMessage::Fail() at ??:0
@ 0x7f986ec0388a google::LogMessage::SendToLog() at ??:0
@ 0x7f986ec004de google::LogMessage::Flush() at ??:0
@ 0x7f986ec01a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f98729647d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f987295373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f988462f1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f988462e783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f98846ba2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f98846ba0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f98847045e9 std::function<>::operator()() at ??:0
@ 0x7f9884632cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9884630c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f98846b8610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f98846b8439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f987184c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f987184f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f987184f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f987182aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9874b046db start_thread at ??:0
@ 0x7f986da9a61f clone at ??:0
W20260619 15:07:13.304080 9153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:13.304461 9153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:13.304638 9153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:13.312637 9153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:13.312813 9153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:13.313002 9153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:07:13.323563 9153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:34271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=36543
--tserver_master_addrs=127.24.103.126:34375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:13.325022 9153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:13.326431 9153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:13.339576 9199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:13.340512 9200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:13.341760 9202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:13.342239 9153 server_base.cc:1034] running on GCE node
I20260619 15:07:13.343063 9153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:13.344012 9153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:13.345213 9153 hybrid_clock.cc:648] HybridClock initialized: now 1781881633345167 us; error 33 us; skew 500 ppm
I20260619 15:07:13.345561 9153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:13 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881633, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:13.348922 9153 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:07:13.350549 9153 webserver.cc:460] Webserver started at http://127.24.103.65:36543/ using document root <none> and password file <none>
I20260619 15:07:13.351372 9153 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:13.351481 9153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:13.357357 9153 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 15:07:13.360356 9209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:13.361805 9153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:07:13.362006 9153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "70664d89ba3545e5bbcf62a84f8c49b9"
format_stamp: "Formatted at 2026-06-19 15:06:27 on dist-test-slave-zpbw"
server_key: "86093e31dd646081e00156c208f3afbe"
server_key_iv: "48b4b49e6162dfd9c6fd5b6363022554"
server_key_version: "encryptionkey@0"
I20260619 15:07:13.363162 9153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:13.379913 9153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:13.384162 9153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:13.384466 9153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:13.385511 9153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:13.387015 9153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:13.387135 9153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:13.387271 9153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:13.387375 9153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:13.422338 9153 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:34271
I20260619 15:07:13.422402 9321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:34271 every 8 connection(s)
I20260619 15:07:13.424018 9153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:07:13.424571 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 9153
I20260619 15:07:13.425544 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:43261
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:34375
--webserver_port=38551
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:07:13.430114 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:13.427880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:33669)
0619 15:07:13.428807 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:13.429093 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":388,"threads_started":1}
W20260619 15:07:13.431145 9322 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:34375 (0 consecutive failures): Network error: Failed to ping master at 127.24.103.126:34375: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
I20260619 15:07:13.432195 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:13.431807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:52231)
0619 15:07:13.431995 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:13.432076 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 15:07:13.433352 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:13.432829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:44843)
0619 15:07:13.433045 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:13.433163 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 15:07:13.433650 9322 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 15:07:13.777225 9325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:13.777603 9325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:13.777786 9325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:13.785545 9325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:13.785748 9325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:13.785945 9325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:07:13.796212 9325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:43261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=38551
--tserver_master_addrs=127.24.103.126:34375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:13.797688 9325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:13.799166 9325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:13.813612 9333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:13.813716 9331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:13.814334 9330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:13.814354 9325 server_base.cc:1034] running on GCE node
I20260619 15:07:13.815251 9325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:13.816334 9325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:13.817597 9325 hybrid_clock.cc:648] HybridClock initialized: now 1781881633817526 us; error 70 us; skew 500 ppm
I20260619 15:07:13.817960 9325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:13 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881633, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:13.821527 9325 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:07:13.823246 9325 webserver.cc:460] Webserver started at http://127.24.103.66:38551/ using document root <none> and password file <none>
I20260619 15:07:13.824040 9325 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:13.824201 9325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:13.830090 9325 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 15:07:13.833279 9340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:13.834885 9325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:07:13.835101 9325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "403c75060d464222b2a712e98303e3d3"
format_stamp: "Formatted at 2026-06-19 15:06:27 on dist-test-slave-zpbw"
server_key: "ab8c58584ba836841ee0c9df5444b7f4"
server_key_iv: "70bc673bfa569ed8ebfdd17872d915a3"
server_key_version: "encryptionkey@0"
I20260619 15:07:13.836189 9325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:13.853018 9325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:13.858038 9325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:13.858359 9325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:13.859516 9325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:13.861003 9325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:13.861100 9325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:13.861181 9325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:13.861250 9325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:13.895830 9325 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:43261
I20260619 15:07:13.895917 9452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:43261 every 8 connection(s)
I20260619 15:07:13.897421 9325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:07:13.902420 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:13.900588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:60517)
0619 15:07:13.901299 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:13.901542 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":285,"threads_started":1}
W20260619 15:07:13.903122 9453 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:34375 (0 consecutive failures): Network error: Failed to ping master at 127.24.103.126:34375: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
I20260619 15:07:13.903234 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 9325
I20260619 15:07:13.904057 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:45535
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:34375
--webserver_port=39001
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:07:13.904112 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:13.903781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:57231)
0619 15:07:13.903929 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:13.904001 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 15:07:13.905705 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:13.904619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:54679)
0619 15:07:13.905228 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:13.905421 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":256,"threads_started":1}
W20260619 15:07:13.905941 9453 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 15:07:14.250985 9457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:14.251295 9457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:14.251394 9457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:14.259131 9457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:14.259256 9457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:14.259372 9457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:07:14.269768 9457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:45535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=39001
--tserver_master_addrs=127.24.103.126:34375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:14.271080 9457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:14.272342 9457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:14.285606 9462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:14.285681 9463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:14.285655 9465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:14.286998 9457 server_base.cc:1034] running on GCE node
I20260619 15:07:14.287679 9457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:14.288652 9457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:14.289849 9457 hybrid_clock.cc:648] HybridClock initialized: now 1781881634289796 us; error 43 us; skew 500 ppm
I20260619 15:07:14.290132 9457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:14 dist-test-slave-zpbw krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881634, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:14.293252 9457 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:07:14.294864 9457 webserver.cc:460] Webserver started at http://127.24.103.67:39001/ using document root <none> and password file <none>
I20260619 15:07:14.295590 9457 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:14.295728 9457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:14.301338 9457 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 15:07:14.304220 9472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:14.305652 9457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:07:14.305840 9457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "961f6fae5e564d9a90ef4ce887500379"
format_stamp: "Formatted at 2026-06-19 15:06:28 on dist-test-slave-zpbw"
server_key: "19673095930822a7026bf497f3b6df18"
server_key_iv: "5bb97b51dbd10eb9d45466db67a1a019"
server_key_version: "encryptionkey@0"
I20260619 15:07:14.306900 9457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:14.328925 9457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:14.333153 9457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:14.333457 9457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:14.334481 9457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:14.335922 9457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:14.336050 9457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:14.336190 9457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:14.336288 9457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:14.371279 9457 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:45535
I20260619 15:07:14.371353 9584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:45535 every 8 connection(s)
I20260619 15:07:14.372906 9457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:07:14.378063 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:14.376182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:40909)
0619 15:07:14.376927 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:14.377174 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":291,"threads_started":1}
W20260619 15:07:14.378863 9585 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:34375 (0 consecutive failures): Network error: Failed to ping master at 127.24.103.126:34375: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
I20260619 15:07:14.379820 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:14.379483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:56241)
0619 15:07:14.379637 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:14.379708 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 15:07:14.380414 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 9457
I20260619 15:07:14.381560 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:14.381130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:40449)
0619 15:07:14.381329 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:14.381429 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260619 15:07:14.381939 9585 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 15:07:14.435286 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:14.434372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:59205)
0619 15:07:14.434917 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:14.435049 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":232,"threads_started":1}
I20260619 15:07:14.907366 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:14.906500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:34969)
0619 15:07:14.907079 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:14.907162 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":340,"threads_started":1}
I20260619 15:07:15.383365 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:15.382472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:32851)
0619 15:07:15.383037 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:15.383128 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":241,"threads_started":1}
I20260619 15:07:15.437140 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:15.436346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:53669)
0619 15:07:15.436877 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:15.436971 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":258,"threads_started":1}
I20260619 15:07:15.909142 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:15.908432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:41475)
0619 15:07:15.908884 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:15.908976 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":238,"threads_started":1}
I20260619 15:07:16.385069 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:16.384354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:39411)
0619 15:07:16.384804 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:16.384895 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":218,"threads_started":1}
I20260619 15:07:16.438943 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:16.438096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:49183)
0619 15:07:16.438586 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:16.438677 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":257,"threads_started":1}
I20260619 15:07:16.911103 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:16.910212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:32927)
0619 15:07:16.910797 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:16.910915 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":283,"threads_started":1}
I20260619 15:07:17.386919 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:17.386062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:47437)
0619 15:07:17.386537 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:17.386630 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":239,"threads_started":1}
I20260619 15:07:17.440691 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:17.439922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:37905)
0619 15:07:17.440393 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:17.440486 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":216,"threads_started":1}
I20260619 15:07:17.912801 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:17.912064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:39241)
0619 15:07:17.912524 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:17.912617 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":226,"threads_started":1}
I20260619 15:07:18.388806 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:18.388047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:57501)
0619 15:07:18.388516 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:18.388609 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":229,"threads_started":1}
I20260619 15:07:18.442405 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:18.441663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:55751)
0619 15:07:18.442147 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:18.442235 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":261,"threads_started":1}
I20260619 15:07:18.914556 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:18.913796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:41561)
0619 15:07:18.914269 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:18.914360 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":231,"threads_started":1}
I20260619 15:07:19.390551 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:19.389728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:55925)
0619 15:07:19.390296 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:19.390380 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":329,"threads_started":1}
I20260619 15:07:19.444283 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:19.443525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:44483)
0619 15:07:19.444011 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:19.444113 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":244,"threads_started":1}
I20260619 15:07:19.916216 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:19.915532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:43403)
0619 15:07:19.915968 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:19.916056 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":220,"threads_started":1}
I20260619 15:07:20.392534 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:20.391647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:41283)
0619 15:07:20.392182 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:20.392357 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":253,"threads_started":1}
I20260619 15:07:20.446164 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:20.445374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:41859)
0619 15:07:20.445862 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:20.445948 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":222,"threads_started":1}
I20260619 15:07:20.917987 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:20.917168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:49301)
0619 15:07:20.917669 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:20.917756 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":224,"threads_started":1}
I20260619 15:07:21.394348 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:21.393558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:59381)
0619 15:07:21.394045 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:21.394134 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":218,"threads_started":1}
I20260619 15:07:21.447995 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:21.447294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:51809)
0619 15:07:21.447754 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:21.447836 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":241,"threads_started":1}
I20260619 15:07:21.919890 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:21.919063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:40323)
0619 15:07:21.919604 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:21.919705 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":275,"threads_started":1}
I20260619 15:07:22.396180 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:22.395447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:37611)
0619 15:07:22.395926 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:22.396018 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":242,"threads_started":1}
I20260619 15:07:22.449801 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:22.449026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:54153)
0619 15:07:22.449526 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:22.449622 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":243,"threads_started":1}
I20260619 15:07:22.921501 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:22.920773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:58309)
0619 15:07:22.921225 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:22.921330 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":233,"threads_started":1}
I20260619 15:07:23.397965 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:23.397180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:60815)
0619 15:07:23.397669 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:23.397766 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":225,"threads_started":1}
I20260619 15:07:23.451882 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:23.450961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:44869)
0619 15:07:23.451560 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:23.451674 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":308,"threads_started":1}
I20260619 15:07:23.923427 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:23.922660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:55519)
0619 15:07:23.923163 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:23.923253 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":296,"threads_started":1}
I20260619 15:07:24.399957 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:24.399090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:43067)
0619 15:07:24.399635 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:24.399747 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":279,"threads_started":1}
I20260619 15:07:24.453677 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:24.452837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:46571)
0619 15:07:24.453389 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:24.453493 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":292,"threads_started":1}
I20260619 15:07:24.925132 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:24.924342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:47451)
0619 15:07:24.924844 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:24.924944 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":263,"threads_started":1}
I20260619 15:07:25.401628 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:25.400889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:50283)
0619 15:07:25.401312 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:25.401431 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":204,"threads_started":1}
I20260619 15:07:25.455333 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:25.454616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:37165)
0619 15:07:25.455093 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:25.455177 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":275,"threads_started":1}
I20260619 15:07:25.926862 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:25.926119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:44371)
0619 15:07:25.926548 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:25.926631 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":229,"threads_started":1}
I20260619 15:07:26.403537 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:26.402675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:45689)
0619 15:07:26.403278 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:26.403379 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":383,"threads_started":1}
I20260619 15:07:26.457073 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:26.456277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:48531)
0619 15:07:26.456771 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:26.456876 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":229,"threads_started":1}
I20260619 15:07:26.928640 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:26.927838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:44799)
0619 15:07:26.928359 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:26.928450 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":246,"threads_started":1}
I20260619 15:07:27.405366 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:27.404590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:41213)
0619 15:07:27.405078 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:27.405180 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":237,"threads_started":1}
I20260619 15:07:27.458966 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:27.458030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:43185)
0619 15:07:27.458595 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:27.458701 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":309,"threads_started":1}
I20260619 15:07:27.930402 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:27.929678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:49431)
0619 15:07:27.930122 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:27.930205 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":215,"threads_started":1}
I20260619 15:07:28.407171 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:28.406343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:49801)
0619 15:07:28.406874 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:28.406978 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260619 15:07:28.460731 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:28.459997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.65:44089)
0619 15:07:28.460449 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:28.460549 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":241,"threads_started":1}
I20260619 15:07:28.932392 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:28.931609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.66:52091)
0619 15:07:28.932085 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:28.932186 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":260,"threads_started":1}
I20260619 15:07:29.385048 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 9153
I20260619 15:07:29.409251 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:29.408486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34375 (local address 127.24.103.67:59227)
0619 15:07:29.408969 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:29.409086 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":278,"threads_started":1}
I20260619 15:07:29.409251 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:29.386413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.24.103.65:34271)
0619 15:07:29.387141 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:29.387161 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:29.387226 (+ 65us) server_negotiation.cc:407] Connection header received
0619 15:07:29.389059 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:29.389070 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:29.389544 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:29.389860 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:29.391477 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:29.392756 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:29.394059 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:29.394896 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:29.395048 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:29.402692 (+ 7644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:29.402702 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:29.402722 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:29.402834 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:29.405823 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:29.406500 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:29.406507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:29.406514 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:29.406687 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:29.407086 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:29.407089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:29.407091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:29.407647 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:29.407877 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:29.408448 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 15:07:29.408966 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":336,"threads_started":1}
I20260619 15:07:29.410908 9317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:29.497193 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 9153
I20260619 15:07:29.515095 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 9325
I20260619 15:07:29.533314 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:29.516131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.24.103.66:43261)
0619 15:07:29.516690 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:29.516704 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:29.516741 (+ 37us) server_negotiation.cc:407] Connection header received
0619 15:07:29.518316 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:29.518325 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:29.518854 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:29.519165 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:29.520909 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:29.522162 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:29.523580 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:29.524286 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:29.524446 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:29.527136 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:29.527145 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:29.527163 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:29.527206 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:29.530249 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:29.531055 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:29.531062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:29.531069 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:29.531225 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:29.531562 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:29.531565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:29.531567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:29.532092 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:29.532332 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:29.532682 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 15:07:29.533013 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":277,"threads_started":1}
I20260619 15:07:29.534677 9448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:29.619956 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 9325
I20260619 15:07:29.638164 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 9457
I20260619 15:07:29.657459 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:29.639153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.24.103.67:45535)
0619 15:07:29.639780 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:29.639793 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:29.639836 (+ 43us) server_negotiation.cc:407] Connection header received
0619 15:07:29.641463 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:29.641473 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:29.641921 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:29.642289 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:29.644055 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:29.645335 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:29.646690 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:29.647492 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:29.647708 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:29.650516 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:29.650525 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:29.650543 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:29.650587 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:29.653700 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:29.654525 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:29.654533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:29.654540 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:29.654719 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:29.655269 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:29.655277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:29.655281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:29.655856 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:29.656175 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:29.656682 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 15:07:29.657154 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":276,"threads_started":1}
I20260619 15:07:29.658952 9580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:29.744436 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 9457
2026-06-19T15:07:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64740 ms)
[----------] 30 tests from SecurityITest (298489 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:07:29 dist-test-slave-zpbw krb5kdc[9643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:07:29 dist-test-slave-zpbw krb5kdc[9643](info): set up 2 sockets
Jun 19 15:07:29 dist-test-slave-zpbw krb5kdc[9643](info): commencing operation
krb5kdc: starting...
W20260619 15:07:30.867223 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:07:31 dist-test-slave-zpbw krb5kdc[9643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:07:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:07:31.142601 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:36049
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:36049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:31.480060 9656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:31.480405 9656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:31.480585 9656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:31.488127 9656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:07:31.488257 9656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:31.488379 9656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:31.488476 9656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:07:31.488554 9656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:07:31.498754 9656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:36049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:36049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:31.500231 9656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:31.501624 9656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:31.514564 9661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:31.514617 9664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:31.514603 9662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:31.515347 9656 server_base.cc:1034] running on GCE node
I20260619 15:07:31.516208 9656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:31.517827 9656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:31.519066 9656 hybrid_clock.cc:648] HybridClock initialized: now 1781881651519012 us; error 44 us; skew 500 ppm
I20260619 15:07:31.519447 9656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:31 dist-test-slave-zpbw krb5kdc[9643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881651, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:31.522600 9656 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:07:31.524266 9656 webserver.cc:460] Webserver started at http://127.24.103.126:38919/ using document root <none> and password file <none>
I20260619 15:07:31.525002 9656 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:31.525151 9656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:31.525465 9656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:31.528823 9656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "44622be5bf754a18ac4f91ba53685629"
format_stamp: "Formatted at 2026-06-19 15:07:31 on dist-test-slave-zpbw"
server_key: "8c99453a19e4fda7163ee56a2a0344a2"
server_key_iv: "fbf166d57f0a33026ebfc09c60c3966d"
server_key_version: "encryptionkey@0"
I20260619 15:07:31.529639 9656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "44622be5bf754a18ac4f91ba53685629"
format_stamp: "Formatted at 2026-06-19 15:07:31 on dist-test-slave-zpbw"
server_key: "8c99453a19e4fda7163ee56a2a0344a2"
server_key_iv: "fbf166d57f0a33026ebfc09c60c3966d"
server_key_version: "encryptionkey@0"
I20260619 15:07:31.535280 9656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:07:31.538969 9671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:31.540467 9656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:07:31.540684 9656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "44622be5bf754a18ac4f91ba53685629"
format_stamp: "Formatted at 2026-06-19 15:07:31 on dist-test-slave-zpbw"
server_key: "8c99453a19e4fda7163ee56a2a0344a2"
server_key_iv: "fbf166d57f0a33026ebfc09c60c3966d"
server_key_version: "encryptionkey@0"
I20260619 15:07:31.540917 9656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:31.559572 9656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:31.563885 9656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:31.564201 9656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:31.583638 9656 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:36049
I20260619 15:07:31.583645 9722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:36049 every 8 connection(s)
I20260619 15:07:31.585144 9656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:07:31.588385 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 9656
I20260619 15:07:31.588627 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:07:31.589478 24989 external_mini_cluster.cc:1428] Setting key a6b36f1033ced78d3c14cf4000296e88
I20260619 15:07:31.589824 9723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:31.604729 9723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629: Bootstrap starting.
Jun 19 15:07:31 dist-test-slave-zpbw krb5kdc[9643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:07:31.608484 9723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:31.609764 9723 log.cc:824] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:31.612834 9723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629: No bootstrap required, opened a new log
I20260619 15:07:31.615310 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:31.591823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.24.103.126:36049)
0619 15:07:31.592664 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:31.592676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:31.592720 (+ 44us) server_negotiation.cc:407] Connection header received
0619 15:07:31.595031 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:31.595060 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:31.595541 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:31.596005 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:31.597619 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:31.598968 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:31.601153 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:31.601922 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:31.602075 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:31.607131 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:31.607140 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:31.607155 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:31.607200 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:31.610510 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:31.611394 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:31.611402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:31.611408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:31.611581 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:31.611992 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:31.611996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:31.611998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:31.612529 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:31.612860 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:31.613359 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 15:07:31.613857 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":376,"threads_started":1}
I20260619 15:07:31.621559 9723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44622be5bf754a18ac4f91ba53685629" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36049 } }
I20260619 15:07:31.621955 9723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:31.622036 9723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44622be5bf754a18ac4f91ba53685629, State: Initialized, Role: FOLLOWER
I20260619 15:07:31.622558 9723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44622be5bf754a18ac4f91ba53685629" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36049 } }
I20260619 15:07:31.622756 9723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:07:31.622941 9723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:07:31.623072 9723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:31.625278 9723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44622be5bf754a18ac4f91ba53685629" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36049 } }
I20260619 15:07:31.625787 9723 leader_election.cc:302] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44622be5bf754a18ac4f91ba53685629; no voters:
I20260619 15:07:31.626428 9723 leader_election.cc:288] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:07:31.626581 9728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:31.627826 9728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [term 1 LEADER]: Becoming Leader. State: Replica: 44622be5bf754a18ac4f91ba53685629, State: Running, Role: LEADER
I20260619 15:07:31.628450 9728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44622be5bf754a18ac4f91ba53685629" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36049 } }
I20260619 15:07:31.628940 9723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:07:31.631633 9730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44622be5bf754a18ac4f91ba53685629. Latest consensus state: current_term: 1 leader_uuid: "44622be5bf754a18ac4f91ba53685629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44622be5bf754a18ac4f91ba53685629" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36049 } } }
I20260619 15:07:31.632161 9730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:31.633312 9729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44622be5bf754a18ac4f91ba53685629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44622be5bf754a18ac4f91ba53685629" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36049 } } }
I20260619 15:07:31.633702 9729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:31.639338 9735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:07:31.644465 9735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:07:31.653086 9735 catalog_manager.cc:1347] Generated new cluster ID: ac5c38c4dc474fe696bb97807cd4f1f4
I20260619 15:07:31.653240 9735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:07:31.669792 9735 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:07:31.671859 9735 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:07:31.682425 9735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629: Generated new TSK 0
I20260619 15:07:31.683683 9735 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:07:31.739509 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36049
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:32.093927 9749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:32.094228 9749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:32.094327 9749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:32.101711 9749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:32.101874 9749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:32.101994 9749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:07:32.112509 9749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:32.113826 9749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:32.115275 9749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:32.128871 9757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:32.128912 9754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:32.128943 9755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:32.130017 9749 server_base.cc:1034] running on GCE node
I20260619 15:07:32.130663 9749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:32.131667 9749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:32.132921 9749 hybrid_clock.cc:648] HybridClock initialized: now 1781881652132854 us; error 66 us; skew 500 ppm
I20260619 15:07:32.133282 9749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:32 dist-test-slave-zpbw krb5kdc[9643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881652, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:32.136778 9749 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:07:32.138283 9749 webserver.cc:460] Webserver started at http://127.24.103.65:39785/ using document root <none> and password file <none>
I20260619 15:07:32.139097 9749 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:32.139236 9749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:32.139597 9749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:32.142740 9749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "8adb903c3bb94560bb824d4ea4339722"
format_stamp: "Formatted at 2026-06-19 15:07:32 on dist-test-slave-zpbw"
server_key: "d33c44a2ad739253136ca59f40d7db8d"
server_key_iv: "4b2e59c977edbcafa1c6ab220f1e49f0"
server_key_version: "encryptionkey@0"
I20260619 15:07:32.143633 9749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "8adb903c3bb94560bb824d4ea4339722"
format_stamp: "Formatted at 2026-06-19 15:07:32 on dist-test-slave-zpbw"
server_key: "d33c44a2ad739253136ca59f40d7db8d"
server_key_iv: "4b2e59c977edbcafa1c6ab220f1e49f0"
server_key_version: "encryptionkey@0"
I20260619 15:07:32.149581 9749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:07:32.153427 9764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:32.155134 9749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 15:07:32.155337 9749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "8adb903c3bb94560bb824d4ea4339722"
format_stamp: "Formatted at 2026-06-19 15:07:32 on dist-test-slave-zpbw"
server_key: "d33c44a2ad739253136ca59f40d7db8d"
server_key_iv: "4b2e59c977edbcafa1c6ab220f1e49f0"
server_key_version: "encryptionkey@0"
I20260619 15:07:32.155565 9749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:32.204109 9749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:32.208316 9749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:32.208619 9749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:32.209640 9749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:32.211099 9749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:32.211175 9749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:32.211328 9749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:32.211380 9749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:32.245645 9749 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:44513
I20260619 15:07:32.245692 9876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:44513 every 8 connection(s)
I20260619 15:07:32.247103 9749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:07:32.256080 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 9749
I20260619 15:07:32.256313 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:07:32.256960 24989 external_mini_cluster.cc:1428] Setting key f9166e888759b87939468fb56afdf1a7
Jun 19 15:07:32 dist-test-slave-zpbw krb5kdc[9643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881652, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:07:32.270118 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:32.249771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:43109 (local address 127.24.103.126:36049)
0619 15:07:32.250213 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:32.250218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:32.251294 (+ 1076us) server_negotiation.cc:407] Connection header received
0619 15:07:32.253229 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:32.253235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:32.253333 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:32.253466 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:32.255208 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:32.256226 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:32.257989 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:32.258431 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:32.258544 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:32.263588 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:32.263594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:32.263597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:32.263638 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:32.266342 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:32.267529 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:32.267536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:32.267541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:32.267711 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:32.268143 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:32.268149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:32.268154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:32.268564 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:32.268700 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:32.269655 (+ 955us) server_negotiation.cc:299] Negotiation successful
0619 15:07:32.269856 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260619 15:07:32.271798 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:32.250356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36049 (local address 127.24.103.65:43109)
0619 15:07:32.251115 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:32.251161 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:32.252918 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:32.253730 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:32.253739 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:32.254300 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:32.254995 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:32.255015 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:32.256465 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:32.256471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:32.257850 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:32.257864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:32.258736 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:32.258745 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:32.259109 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:32.259892 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:07:32.259926 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:32.263285 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:32.266536 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:32.266550 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:32.266573 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:32.267356 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:32.267827 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:32.267831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:32.267833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:32.268026 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:32.268710 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:32.268720 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:32.269229 (+ 509us) client_negotiation.cc:769] Sending connection context
0619 15:07:32.269561 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 15:07:32.270190 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":275,"threads_started":1}
I20260619 15:07:32.273468 9877 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36049
I20260619 15:07:32.273789 9877 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:32.274541 9877 heartbeater.cc:511] Master 127.24.103.126:36049 requested a full tablet report, sending...
I20260619 15:07:32.276822 9688 ts_manager.cc:194] Registered new tserver with Master: 8adb903c3bb94560bb824d4ea4339722 (127.24.103.65:44513)
I20260619 15:07:32.278328 9688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:43109
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:07:32.313620 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36049
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:32.657083 9883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:32.657431 9883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:32.657599 9883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:32.665125 9883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:32.665307 9883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:32.665479 9883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:07:32.675508 9883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:32.676988 9883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:32.678364 9883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:32.692741 9888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:32.692807 9889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:32.693023 9891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:32.693789 9883 server_base.cc:1034] running on GCE node
I20260619 15:07:32.694602 9883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:32.695693 9883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:32.696950 9883 hybrid_clock.cc:648] HybridClock initialized: now 1781881652696892 us; error 72 us; skew 500 ppm
I20260619 15:07:32.697309 9883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:32 dist-test-slave-zpbw krb5kdc[9643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881652, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:32.700628 9883 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:07:32.702227 9883 webserver.cc:460] Webserver started at http://127.24.103.66:43371/ using document root <none> and password file <none>
I20260619 15:07:32.703034 9883 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:32.703184 9883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:32.703528 9883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:32.706862 9883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319"
format_stamp: "Formatted at 2026-06-19 15:07:32 on dist-test-slave-zpbw"
server_key: "35a78ca79d9d27ad15f4f91e8b630a93"
server_key_iv: "6206c9e0fd8353dfdb1407710bdd1ddb"
server_key_version: "encryptionkey@0"
I20260619 15:07:32.707703 9883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319"
format_stamp: "Formatted at 2026-06-19 15:07:32 on dist-test-slave-zpbw"
server_key: "35a78ca79d9d27ad15f4f91e8b630a93"
server_key_iv: "6206c9e0fd8353dfdb1407710bdd1ddb"
server_key_version: "encryptionkey@0"
I20260619 15:07:32.713573 9883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:07:32.717996 9898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:32.719576 9883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 15:07:32.719832 9883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319"
format_stamp: "Formatted at 2026-06-19 15:07:32 on dist-test-slave-zpbw"
server_key: "35a78ca79d9d27ad15f4f91e8b630a93"
server_key_iv: "6206c9e0fd8353dfdb1407710bdd1ddb"
server_key_version: "encryptionkey@0"
I20260619 15:07:32.720106 9883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:32.738940 9883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:32.744364 9883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:32.744704 9883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:32.745762 9883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:32.747195 9883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:32.747308 9883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:32.747462 9883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:32.747534 9883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:32.781885 9883 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:45643
I20260619 15:07:32.781927 10010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:45643 every 8 connection(s)
I20260619 15:07:32.783365 9883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:07:32.788609 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 9883
I20260619 15:07:32.788828 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:07:32.789290 24989 external_mini_cluster.cc:1428] Setting key 1f8da68db7b70d873fded334a14920b9
Jun 19 15:07:32 dist-test-slave-zpbw krb5kdc[9643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881652, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:07:32.805212 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:32.786023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:57035 (local address 127.24.103.126:36049)
0619 15:07:32.786482 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:32.786487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:32.787646 (+ 1159us) server_negotiation.cc:407] Connection header received
0619 15:07:32.789646 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:32.789652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:32.789754 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:32.789884 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:32.791527 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:32.792270 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:32.793853 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:32.794311 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:32.794451 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:32.798539 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:32.798544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:32.798548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:32.798584 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:32.801339 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:32.802184 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:32.802189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:32.802191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:32.802513 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:32.803264 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:32.803268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:32.803270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:32.803565 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:32.803740 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:32.804563 (+ 823us) server_negotiation.cc:299] Negotiation successful
0619 15:07:32.804810 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":205,"threads_started":1}
I20260619 15:07:32.806227 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:32.786669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36049 (local address 127.24.103.66:57035)
0619 15:07:32.787473 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:32.787513 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:32.789315 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:32.790137 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:32.790146 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:32.790686 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:32.791351 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:32.791369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:32.792432 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:32.792438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:32.793699 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:32.793711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:32.794563 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:32.794570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:32.795089 (+ 519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:32.795900 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:07:32.795929 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:32.798257 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:32.801512 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:32.801522 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:32.801542 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:32.802020 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:32.802676 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:32.802682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:32.802686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:32.802988 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:32.803729 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:32.803737 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:32.804136 (+ 399us) client_negotiation.cc:769] Sending connection context
0619 15:07:32.804442 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 15:07:32.805036 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":288,"threads_started":1}
I20260619 15:07:32.807854 10011 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36049
I20260619 15:07:32.808210 10011 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:32.809137 10011 heartbeater.cc:511] Master 127.24.103.126:36049 requested a full tablet report, sending...
I20260619 15:07:32.811081 9688 ts_manager.cc:194] Registered new tserver with Master: c86e9cbba6c240a3b1f30d8fcd9ab319 (127.24.103.66:45643)
I20260619 15:07:32.812444 9688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:57035
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:07:32.846371 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36049
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:33.191658 10017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:33.191977 10017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:33.192073 10017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:33.199584 10017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:33.199713 10017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:33.199913 10017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:07:33.209913 10017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:33.211309 10017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:33.212602 10017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:33.226547 10025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:33.226615 10022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:33.226639 10023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:33.227759 10017 server_base.cc:1034] running on GCE node
I20260619 15:07:33.228453 10017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:33.229607 10017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:33.230871 10017 hybrid_clock.cc:648] HybridClock initialized: now 1781881653230822 us; error 43 us; skew 500 ppm
I20260619 15:07:33.231212 10017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:33 dist-test-slave-zpbw krb5kdc[9643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881653, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:33.234548 10017 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:07:33.236418 10017 webserver.cc:460] Webserver started at http://127.24.103.67:33093/ using document root <none> and password file <none>
I20260619 15:07:33.237185 10017 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:33.237344 10017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:33.237679 10017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:33.240783 10017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "38c91539aa404d77b3091d3b86cf721a"
format_stamp: "Formatted at 2026-06-19 15:07:33 on dist-test-slave-zpbw"
server_key: "9b6fa8d3cd4035d25d950671d11001ed"
server_key_iv: "0b3b5b51f2e37fecd70e0063faeaae30"
server_key_version: "encryptionkey@0"
I20260619 15:07:33.241596 10017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "38c91539aa404d77b3091d3b86cf721a"
format_stamp: "Formatted at 2026-06-19 15:07:33 on dist-test-slave-zpbw"
server_key: "9b6fa8d3cd4035d25d950671d11001ed"
server_key_iv: "0b3b5b51f2e37fecd70e0063faeaae30"
server_key_version: "encryptionkey@0"
I20260619 15:07:33.247838 10017 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260619 15:07:33.251824 10032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:33.253304 10017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:07:33.253475 10017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "38c91539aa404d77b3091d3b86cf721a"
format_stamp: "Formatted at 2026-06-19 15:07:33 on dist-test-slave-zpbw"
server_key: "9b6fa8d3cd4035d25d950671d11001ed"
server_key_iv: "0b3b5b51f2e37fecd70e0063faeaae30"
server_key_version: "encryptionkey@0"
I20260619 15:07:33.253671 10017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:33.268231 10017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:33.273073 10017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:33.273337 10017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:33.274309 10017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:33.275805 10017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:33.275883 10017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:33.275971 10017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:33.276021 10017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:33.281597 9877 heartbeater.cc:503] Master 127.24.103.126:36049 was elected leader, sending a full tablet report...
I20260619 15:07:33.311525 10017 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:37577
I20260619 15:07:33.311564 10144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:37577 every 8 connection(s)
I20260619 15:07:33.313052 10017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:07:33.322322 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 10017
I20260619 15:07:33.322516 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:07:33.323035 24989 external_mini_cluster.cc:1428] Setting key b14582f9e76a1ff877bf2c5bfb3a2bc7
Jun 19 15:07:33 dist-test-slave-zpbw krb5kdc[9643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881653, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:07:33.335381 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.315751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:32919 (local address 127.24.103.126:36049)
0619 15:07:33.316266 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:33.316270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:33.317265 (+ 995us) server_negotiation.cc:407] Connection header received
0619 15:07:33.319331 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:33.319337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:33.319458 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:33.319594 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:33.321289 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.322138 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.324400 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.324863 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.325050 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.329380 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:33.329385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:33.329388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:33.329423 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:33.332120 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:33.332925 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:33.332930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:33.332933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:33.333060 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:33.333517 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:33.333522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:33.333524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:33.333798 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:33.333964 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:33.334683 (+ 719us) server_negotiation.cc:299] Negotiation successful
0619 15:07:33.334972 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":245,"threads_started":1}
I20260619 15:07:33.336414 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.316304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36049 (local address 127.24.103.67:32919)
0619 15:07:33.317072 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:33.317128 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:33.319025 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:33.319850 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:33.319858 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:33.320423 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:33.321086 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.321125 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.322308 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.322316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.324161 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.324175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.325518 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.325523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.325761 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.326634 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:07:33.326661 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:33.329130 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:33.332285 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:33.332295 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:33.332314 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:33.332771 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:33.333173 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:33.333178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:33.333182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:33.333388 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:33.333929 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:33.333938 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:33.334307 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 15:07:33.334580 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 15:07:33.335190 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":301,"threads_started":1}
I20260619 15:07:33.338089 10145 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36049
I20260619 15:07:33.338444 10145 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:33.339339 10145 heartbeater.cc:511] Master 127.24.103.126:36049 requested a full tablet report, sending...
I20260619 15:07:33.340780 9688 ts_manager.cc:194] Registered new tserver with Master: 38c91539aa404d77b3091d3b86cf721a (127.24.103.67:37577)
I20260619 15:07:33.341581 9688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:32919
I20260619 15:07:33.351895 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:07:33.370009 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.356756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34820 (local address 127.24.103.126:36049)
0619 15:07:33.357012 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:33.357021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:33.357431 (+ 410us) server_negotiation.cc:407] Connection header received
0619 15:07:33.357645 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:33.357649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:33.357723 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:33.357813 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:33.359492 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.360363 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.361821 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.362237 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.362340 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.364786 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:33.364791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:33.364794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:33.364831 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:33.367375 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:33.368110 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:33.368114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:33.368116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:33.368233 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:33.368551 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:33.368554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:33.368556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:33.368879 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:33.369065 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:33.369410 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 15:07:33.369595 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:07:33.374336 9688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:07:33.381542 9688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:07:33.420949 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.395287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.24.103.65:44513)
0619 15:07:33.407691 (+ 12404us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:33.407705 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:33.409036 (+ 1331us) server_negotiation.cc:407] Connection header received
0619 15:07:33.409591 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:33.409605 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:33.409994 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:33.410208 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:33.411523 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.413589 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.416910 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.418532 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.418701 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.419018 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:33.420396 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0619 15:07:33.420600 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12131,"thread_start_us":12447,"threads_started":1}
I20260619 15:07:33.420994 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.394978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44513 (local address 127.0.0.1:46948)
0619 15:07:33.408718 (+ 13740us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:33.408783 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:33.409275 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:33.410636 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:33.410648 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:33.410693 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:33.411286 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.411305 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.413836 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.413845 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.416745 (+ 2900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.416758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.419170 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.419177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.419415 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.420150 (+ 735us) client_negotiation.cc:769] Sending connection context
0619 15:07:33.420297 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 15:07:33.420521 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13377,"mutex_wait_us":626,"thread_start_us":13212,"threads_started":1}
I20260619 15:07:33.423633 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.395937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.24.103.67:37577)
0619 15:07:33.415128 (+ 19191us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:33.415139 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:33.415178 (+ 39us) server_negotiation.cc:407] Connection header received
0619 15:07:33.415284 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:33.415292 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:33.415595 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:33.415818 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:33.416678 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.417938 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.420134 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.421829 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.421949 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.422172 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:33.423128 (+ 956us) server_negotiation.cc:299] Negotiation successful
0619 15:07:33.423300 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18991,"thread_start_us":18838,"threads_started":1}
I20260619 15:07:33.424046 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.395305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37577 (local address 127.0.0.1:38814)
0619 15:07:33.414675 (+ 19370us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:33.414705 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:33.414902 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:33.415933 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:33.415938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:33.415965 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:33.416451 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.416468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.418111 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.418116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.419973 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.419986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.422325 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.422331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.422516 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.422551 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 15:07:33.422648 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 15:07:33.422819 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19218,"thread_start_us":19860,"threads_started":1}
I20260619 15:07:33.424618 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.395901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:45643 (local address 127.0.0.1:39868)
0619 15:07:33.415462 (+ 19561us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:33.415480 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:33.415601 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:33.416298 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:33.416302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:33.416332 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:33.416911 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.416928 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.418808 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.418813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.421836 (+ 3023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.421852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.423613 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.423620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.423781 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.423812 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 15:07:33.423902 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 15:07:33.424041 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19441,"thread_start_us":18413,"threads_started":1}
I20260619 15:07:33.424667 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.395968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.24.103.66:45643)
0619 15:07:33.413835 (+ 17867us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:33.413849 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:33.415742 (+ 1893us) server_negotiation.cc:407] Connection header received
0619 15:07:33.415854 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:33.415862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:33.416100 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:33.416313 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:33.417090 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.418610 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.422050 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.423401 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.423565 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.423803 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:33.424158 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 15:07:33.424329 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17613,"thread_start_us":17892,"threads_started":1}
I20260619 15:07:33.425668 9812 tablet_service.cc:1467] Processing CreateTablet for tablet ccb9c0568e9341589b91aafa08fc279a (DEFAULT_TABLE table=test-table [id=acb499a4f85448b287f31d80f16cca50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:33.427290 9812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb9c0568e9341589b91aafa08fc279a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:33.427979 10080 tablet_service.cc:1467] Processing CreateTablet for tablet ccb9c0568e9341589b91aafa08fc279a (DEFAULT_TABLE table=test-table [id=acb499a4f85448b287f31d80f16cca50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:33.428344 9946 tablet_service.cc:1467] Processing CreateTablet for tablet ccb9c0568e9341589b91aafa08fc279a (DEFAULT_TABLE table=test-table [id=acb499a4f85448b287f31d80f16cca50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:33.429550 10080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb9c0568e9341589b91aafa08fc279a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:33.429663 9946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb9c0568e9341589b91aafa08fc279a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:33.439900 10164 tablet_bootstrap.cc:492] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722: Bootstrap starting.
I20260619 15:07:33.443392 10164 tablet_bootstrap.cc:654] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:33.445015 10166 tablet_bootstrap.cc:492] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a: Bootstrap starting.
I20260619 15:07:33.446281 10165 tablet_bootstrap.cc:492] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319: Bootstrap starting.
I20260619 15:07:33.444890 10164 log.cc:824] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:33.449294 10166 tablet_bootstrap.cc:654] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:33.449703 10165 tablet_bootstrap.cc:654] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:33.450619 10164 tablet_bootstrap.cc:492] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722: No bootstrap required, opened a new log
I20260619 15:07:33.450701 10166 log.cc:824] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:33.450719 10165 log.cc:824] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:33.450975 10164 ts_tablet_manager.cc:1397] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260619 15:07:33.454028 10165 tablet_bootstrap.cc:492] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319: No bootstrap required, opened a new log
I20260619 15:07:33.454015 10166 tablet_bootstrap.cc:492] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a: No bootstrap required, opened a new log
I20260619 15:07:33.454330 10165 ts_tablet_manager.cc:1397] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 15:07:33.454459 10166 ts_tablet_manager.cc:1397] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 15:07:33.462646 10166 raft_consensus.cc:348] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8adb903c3bb94560bb824d4ea4339722" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44513 } } peers { permanent_uuid: "38c91539aa404d77b3091d3b86cf721a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37577 } } peers { permanent_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45643 } }
I20260619 15:07:33.462697 10164 raft_consensus.cc:348] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8adb903c3bb94560bb824d4ea4339722" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44513 } } peers { permanent_uuid: "38c91539aa404d77b3091d3b86cf721a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37577 } } peers { permanent_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45643 } }
I20260619 15:07:33.463186 10166 raft_consensus.cc:374] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:33.463210 10164 raft_consensus.cc:374] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:33.463299 10164 raft_consensus.cc:729] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8adb903c3bb94560bb824d4ea4339722, State: Initialized, Role: FOLLOWER
I20260619 15:07:33.463289 10166 raft_consensus.cc:729] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c91539aa404d77b3091d3b86cf721a, State: Initialized, Role: FOLLOWER
I20260619 15:07:33.464006 10166 consensus_queue.cc:260] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8adb903c3bb94560bb824d4ea4339722" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44513 } } peers { permanent_uuid: "38c91539aa404d77b3091d3b86cf721a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37577 } } peers { permanent_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45643 } }
I20260619 15:07:33.464011 10164 consensus_queue.cc:260] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8adb903c3bb94560bb824d4ea4339722" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44513 } } peers { permanent_uuid: "38c91539aa404d77b3091d3b86cf721a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37577 } } peers { permanent_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45643 } }
I20260619 15:07:33.465241 10145 heartbeater.cc:503] Master 127.24.103.126:36049 was elected leader, sending a full tablet report...
I20260619 15:07:33.465521 10165 raft_consensus.cc:348] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8adb903c3bb94560bb824d4ea4339722" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44513 } } peers { permanent_uuid: "38c91539aa404d77b3091d3b86cf721a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37577 } } peers { permanent_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45643 } }
I20260619 15:07:33.465988 10166 ts_tablet_manager.cc:1428] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260619 15:07:33.465988 10165 raft_consensus.cc:374] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:33.466087 10165 raft_consensus.cc:729] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86e9cbba6c240a3b1f30d8fcd9ab319, State: Initialized, Role: FOLLOWER
I20260619 15:07:33.466747 10165 consensus_queue.cc:260] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8adb903c3bb94560bb824d4ea4339722" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44513 } } peers { permanent_uuid: "38c91539aa404d77b3091d3b86cf721a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37577 } } peers { permanent_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45643 } }
I20260619 15:07:33.475242 10011 heartbeater.cc:503] Master 127.24.103.126:36049 was elected leader, sending a full tablet report...
I20260619 15:07:33.475858 10164 ts_tablet_manager.cc:1428] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722: Time spent starting tablet: real 0.025s user 0.021s sys 0.000s
I20260619 15:07:33.475883 10165 ts_tablet_manager.cc:1428] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319: Time spent starting tablet: real 0.021s user 0.010s sys 0.012s
W20260619 15:07:33.499622 9878 tablet.cc:2369] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:07:33.535609 10012 tablet.cc:2369] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:07:33.564904 10146 tablet.cc:2369] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:07:33.779304 10170 raft_consensus.cc:482] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:07:33.779783 10170 raft_consensus.cc:504] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8adb903c3bb94560bb824d4ea4339722" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44513 } } peers { permanent_uuid: "38c91539aa404d77b3091d3b86cf721a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37577 } } peers { permanent_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45643 } }
I20260619 15:07:33.781756 10170 leader_election.cc:288] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8adb903c3bb94560bb824d4ea4339722 (127.24.103.65:44513), c86e9cbba6c240a3b1f30d8fcd9ab319 (127.24.103.66:45643)
I20260619 15:07:33.785712 10171 raft_consensus.cc:482] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:07:33.786089 10171 raft_consensus.cc:504] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8adb903c3bb94560bb824d4ea4339722" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44513 } } peers { permanent_uuid: "38c91539aa404d77b3091d3b86cf721a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37577 } } peers { permanent_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45643 } }
I20260619 15:07:33.790643 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.782574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:55071 (local address 127.24.103.66:45643)
0619 15:07:33.782840 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:33.782849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:33.783304 (+ 455us) server_negotiation.cc:407] Connection header received
0619 15:07:33.783494 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:33.783501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:33.783568 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:33.783618 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:33.784398 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.785450 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.788045 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.789405 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.789532 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.789604 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:33.790233 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 15:07:33.790377 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 15:07:33.791345 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.782201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:45643 (local address 127.24.103.67:55071)
0619 15:07:33.783180 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:33.783198 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:33.783327 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:33.783845 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:33.783849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:33.783874 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:33.784220 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.784231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.785686 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.785691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.787866 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.787881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.789790 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.789795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.790017 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.790061 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 15:07:33.790847 (+ 786us) client_negotiation.cc:240] Negotiation successful
0619 15:07:33.790988 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":688,"threads_started":1}
I20260619 15:07:33.791923 10171 leader_election.cc:288] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38c91539aa404d77b3091d3b86cf721a (127.24.103.67:37577), c86e9cbba6c240a3b1f30d8fcd9ab319 (127.24.103.66:45643)
I20260619 15:07:33.792752 9966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb9c0568e9341589b91aafa08fc279a" candidate_uuid: "38c91539aa404d77b3091d3b86cf721a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" is_pre_election: true
I20260619 15:07:33.793402 9966 raft_consensus.cc:2454] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c91539aa404d77b3091d3b86cf721a in term 0.
I20260619 15:07:33.794468 10035 leader_election.cc:302] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c91539aa404d77b3091d3b86cf721a, c86e9cbba6c240a3b1f30d8fcd9ab319; no voters:
I20260619 15:07:33.795003 10170 raft_consensus.cc:2781] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:07:33.795141 10170 raft_consensus.cc:482] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:07:33.795224 10170 raft_consensus.cc:3037] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:33.799220 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.781949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44513 (local address 127.24.103.67:37353)
0619 15:07:33.782520 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:33.782543 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:33.782678 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:33.783415 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:33.783421 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:33.783454 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:33.783636 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.783646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.785180 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.785186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.787271 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.787291 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.790182 (+ 2891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.790191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.790467 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.790517 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 15:07:33.790633 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 15:07:33.791212 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439}
I20260619 15:07:33.798555 10170 raft_consensus.cc:504] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8adb903c3bb94560bb824d4ea4339722" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44513 } } peers { permanent_uuid: "38c91539aa404d77b3091d3b86cf721a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37577 } } peers { permanent_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45643 } }
I20260619 15:07:33.800190 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.789820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:38059 (local address 127.24.103.67:37577)
0619 15:07:33.790091 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:33.790099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:33.792700 (+ 2601us) server_negotiation.cc:407] Connection header received
0619 15:07:33.793007 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:33.793015 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:33.793094 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:33.793159 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:33.793917 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.795060 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.797738 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.799033 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.799176 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.799254 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:33.799817 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 15:07:33.799952 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 15:07:33.800542 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.792811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:44037 (local address 127.24.103.66:45643)
0619 15:07:33.793000 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:33.793005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:33.793061 (+ 56us) server_negotiation.cc:407] Connection header received
0619 15:07:33.793329 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:33.793339 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:33.793417 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:33.793490 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:33.794296 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.795467 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.797984 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.799344 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.799502 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.799571 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:33.800126 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 15:07:33.800274 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:07:33.800876 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.788464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37577 (local address 127.24.103.65:38059)
0619 15:07:33.792589 (+ 4125us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:33.792617 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:33.792836 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:33.793343 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:33.793347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:33.793373 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:33.793743 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.793753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.795233 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.795238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.797555 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.797570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.799437 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.799443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.799623 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.799657 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 15:07:33.800317 (+ 660us) client_negotiation.cc:240] Negotiation successful
0619 15:07:33.800486 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3979,"thread_start_us":3799,"threads_started":1}
I20260619 15:07:33.801127 10170 leader_election.cc:288] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [CANDIDATE]: Term 1 election: Requested vote from peers 8adb903c3bb94560bb824d4ea4339722 (127.24.103.65:44513), c86e9cbba6c240a3b1f30d8fcd9ab319 (127.24.103.66:45643)
I20260619 15:07:33.801065 9966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb9c0568e9341589b91aafa08fc279a" candidate_uuid: "38c91539aa404d77b3091d3b86cf721a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319"
I20260619 15:07:33.801286 9966 raft_consensus.cc:3037] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:33.801498 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.782294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:37353 (local address 127.24.103.65:44513)
0619 15:07:33.783043 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:33.783049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:33.783068 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:07:33.783145 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:33.783151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:33.783244 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:33.783335 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:33.783841 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.785012 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.787874 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.789317 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.789513 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.789626 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:33.800949 (+ 11323us) server_negotiation.cc:299] Negotiation successful
0619 15:07:33.801267 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595}
I20260619 15:07:33.801980 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.792234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:45643 (local address 127.24.103.65:44037)
0619 15:07:33.792962 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:33.792981 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:33.793179 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:33.793694 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:33.793705 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:33.793740 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:33.794102 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.794113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.795657 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.795662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.797811 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:33.797825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.799779 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.799785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:33.799940 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.799972 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 15:07:33.800715 (+ 743us) client_negotiation.cc:240] Negotiation successful
0619 15:07:33.800854 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":757,"threads_started":1}
I20260619 15:07:33.801832 10100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb9c0568e9341589b91aafa08fc279a" candidate_uuid: "8adb903c3bb94560bb824d4ea4339722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c91539aa404d77b3091d3b86cf721a" is_pre_election: true
I20260619 15:07:33.802301 10100 raft_consensus.cc:2379] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8adb903c3bb94560bb824d4ea4339722 in current term 1: Already voted for candidate 38c91539aa404d77b3091d3b86cf721a in this term.
I20260619 15:07:33.802569 9831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb9c0568e9341589b91aafa08fc279a" candidate_uuid: "38c91539aa404d77b3091d3b86cf721a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8adb903c3bb94560bb824d4ea4339722"
I20260619 15:07:33.802528 9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb9c0568e9341589b91aafa08fc279a" candidate_uuid: "38c91539aa404d77b3091d3b86cf721a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8adb903c3bb94560bb824d4ea4339722" is_pre_election: true
I20260619 15:07:33.803006 9831 raft_consensus.cc:3037] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:33.803658 9965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb9c0568e9341589b91aafa08fc279a" candidate_uuid: "8adb903c3bb94560bb824d4ea4339722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" is_pre_election: true
I20260619 15:07:33.804718 9966 raft_consensus.cc:2454] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c91539aa404d77b3091d3b86cf721a in term 1.
I20260619 15:07:33.804648 9767 leader_election.cc:302] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8adb903c3bb94560bb824d4ea4339722; no voters: 38c91539aa404d77b3091d3b86cf721a, c86e9cbba6c240a3b1f30d8fcd9ab319
I20260619 15:07:33.805166 10035 leader_election.cc:302] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c91539aa404d77b3091d3b86cf721a, c86e9cbba6c240a3b1f30d8fcd9ab319; no voters:
I20260619 15:07:33.805536 10170 raft_consensus.cc:2781] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:33.805976 10170 raft_consensus.cc:686] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 1 LEADER]: Becoming Leader. State: Replica: 38c91539aa404d77b3091d3b86cf721a, State: Running, Role: LEADER
I20260619 15:07:33.806537 9831 raft_consensus.cc:2454] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c91539aa404d77b3091d3b86cf721a in term 1.
I20260619 15:07:33.806875 10171 raft_consensus.cc:2726] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 15:07:33.806919 10170 consensus_queue.cc:237] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8adb903c3bb94560bb824d4ea4339722" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44513 } } peers { permanent_uuid: "38c91539aa404d77b3091d3b86cf721a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37577 } } peers { permanent_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45643 } }
I20260619 15:07:33.811690 9688 catalog_manager.cc:5495] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a reported cstate change: term changed from 0 to 1, leader changed from <none> to 38c91539aa404d77b3091d3b86cf721a (127.24.103.67). New cstate: current_term: 1 leader_uuid: "38c91539aa404d77b3091d3b86cf721a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8adb903c3bb94560bb824d4ea4339722" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38c91539aa404d77b3091d3b86cf721a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45643 } health_report { overall_health: UNKNOWN } } }
I20260619 15:07:33.887046 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.880926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.24.103.67:37577)
0619 15:07:33.881122 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:33.881130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:33.881575 (+ 445us) server_negotiation.cc:407] Connection header received
0619 15:07:33.881831 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:33.881836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:33.881931 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:33.882090 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:07:33.882502 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.883334 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.884820 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.885258 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.885392 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.885899 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:07:33.886415 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:07:33.886572 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:33.886630 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 15:07:33.886710 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 15:07:33.895493 9966 raft_consensus.cc:1261] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319 [term 1 FOLLOWER]: Refusing update from remote peer 38c91539aa404d77b3091d3b86cf721a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:07:33.895496 9831 raft_consensus.cc:1261] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [term 1 FOLLOWER]: Refusing update from remote peer 38c91539aa404d77b3091d3b86cf721a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:07:33.896411 10176 consensus_queue.cc:1035] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8adb903c3bb94560bb824d4ea4339722" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:07:33.896793 10170 consensus_queue.cc:1035] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86e9cbba6c240a3b1f30d8fcd9ab319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:07:33.911777 10180 mvcc.cc:205] Tried to move back new op lower bound from 7298587254345842688 to 7298587253998481408. Current Snapshot: MvccSnapshot[applied={T|T < 7298587254345842688 or (T in {7298587254345842688})}]
I20260619 15:07:33.925518 9688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:07:33.925794 9688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:07:33.929375 9688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44622be5bf754a18ac4f91ba53685629: Sending DeleteTablet for 3 replicas of tablet ccb9c0568e9341589b91aafa08fc279a
I20260619 15:07:33.930649 10080 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb9c0568e9341589b91aafa08fc279a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:33 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:38814
I20260619 15:07:33.930634 9812 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb9c0568e9341589b91aafa08fc279a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:33 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:46948
I20260619 15:07:33.930855 9946 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb9c0568e9341589b91aafa08fc279a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:33 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:39868
I20260619 15:07:33.936883 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 9749
I20260619 15:07:33.946277 10192 tablet_replica.cc:330] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319: stopping tablet replica
I20260619 15:07:33.946841 10192 raft_consensus.cc:2229] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:07:33.947430 10192 raft_consensus.cc:2258] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:33.950269 10190 tablet_replica.cc:330] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722: stopping tablet replica
I20260619 15:07:33.951297 10191 tablet_replica.cc:330] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a: stopping tablet replica
I20260619 15:07:33.951712 10191 raft_consensus.cc:2229] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:07:33.951947 10192 ts_tablet_manager.cc:1905] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:33.952400 10191 raft_consensus.cc:2258] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:33.950727 10190 raft_consensus.cc:2229] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:07:33.953357 10190 raft_consensus.cc:2258] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:33.955410 10190 ts_tablet_manager.cc:1905] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:33.955413 10191 ts_tablet_manager.cc:1905] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 15:07:33 dist-test-slave-zpbw krb5kdc[9643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:07:33.963238 10191 ts_tablet_manager.cc:1918] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:33.963231 10190 ts_tablet_manager.cc:1918] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:33.963414 10190 log.cc:1196] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/ccb9c0568e9341589b91aafa08fc279a
I20260619 15:07:33.963425 10191 log.cc:1196] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/ccb9c0568e9341589b91aafa08fc279a
I20260619 15:07:33.963950 10191 ts_tablet_manager.cc:1939] T ccb9c0568e9341589b91aafa08fc279a P 38c91539aa404d77b3091d3b86cf721a: Deleting consensus metadata
I20260619 15:07:33.963951 10190 ts_tablet_manager.cc:1939] T ccb9c0568e9341589b91aafa08fc279a P 8adb903c3bb94560bb824d4ea4339722: Deleting consensus metadata
I20260619 15:07:33.965833 9672 catalog_manager.cc:4841] TS 8adb903c3bb94560bb824d4ea4339722 (127.24.103.65:44513): tablet ccb9c0568e9341589b91aafa08fc279a (table test-table [id=acb499a4f85448b287f31d80f16cca50]) successfully deleted
I20260619 15:07:33.965804 9673 catalog_manager.cc:4841] TS 38c91539aa404d77b3091d3b86cf721a (127.24.103.67:37577): tablet ccb9c0568e9341589b91aafa08fc279a (table test-table [id=acb499a4f85448b287f31d80f16cca50]) successfully deleted
I20260619 15:07:33.966807 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:33.938736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.24.103.65:44513)
0619 15:07:33.938968 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:33.938975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:33.939902 (+ 927us) server_negotiation.cc:407] Connection header received
0619 15:07:33.940812 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:33.940821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:33.940932 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:33.941024 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:33.945849 (+ 4825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.947079 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.949637 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:33.950359 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:33.950499 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:33.957118 (+ 6619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:33.957142 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:33.957169 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:33.957229 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:33.961876 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:33.962949 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:33.962959 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:33.962963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:33.963148 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:33.963948 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:33.963953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:33.963956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:33.964541 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:33.964856 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:33.965539 (+ 683us) server_negotiation.cc:299] Negotiation successful
0619 15:07:33.966370 (+ 831us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 15:07:33.968250 9872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:33.970044 10192 ts_tablet_manager.cc:1918] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:33.970247 10192 log.cc:1196] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/ccb9c0568e9341589b91aafa08fc279a
I20260619 15:07:33.970839 10192 ts_tablet_manager.cc:1939] T ccb9c0568e9341589b91aafa08fc279a P c86e9cbba6c240a3b1f30d8fcd9ab319: Deleting consensus metadata
I20260619 15:07:33.973563 9674 catalog_manager.cc:4841] TS c86e9cbba6c240a3b1f30d8fcd9ab319 (127.24.103.66:45643): tablet ccb9c0568e9341589b91aafa08fc279a (table test-table [id=acb499a4f85448b287f31d80f16cca50]) successfully deleted
I20260619 15:07:34.064366 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 9749
I20260619 15:07:34.082670 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 9883
Jun 19 15:07:34 dist-test-slave-zpbw krb5kdc[9643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:07:34.099952 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:34.083503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.24.103.66:45643)
0619 15:07:34.083672 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:34.083679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:34.083760 (+ 81us) server_negotiation.cc:407] Connection header received
0619 15:07:34.083978 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:34.083984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:34.084071 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:34.084179 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:34.085457 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:34.086131 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:34.087700 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:34.088130 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:34.088271 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:34.092802 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:34.092823 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:34.092845 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:34.092901 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:34.096815 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:34.097729 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:34.097739 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:34.097744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:34.097907 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:34.098262 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:34.098267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:34.098271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:34.098739 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:34.099106 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:34.099367 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 15:07:34.099609 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 15:07:34.100718 10006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:34.191437 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 9883
I20260619 15:07:34.209129 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 10017
Jun 19 15:07:34 dist-test-slave-zpbw krb5kdc[9643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:07:34.227080 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:34.210092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.24.103.67:37577)
0619 15:07:34.210323 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:34.210331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:34.210358 (+ 27us) server_negotiation.cc:407] Connection header received
0619 15:07:34.210428 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:34.210433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:34.210527 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:34.210652 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:34.212026 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:34.213222 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:34.214513 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:34.215331 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:34.215500 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:34.220294 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:34.220313 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:34.220335 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:34.220394 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:34.224043 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:34.224825 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:34.224832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:34.224834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:34.224969 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:34.225389 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:34.225396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:34.225405 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:34.225820 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:34.226153 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:34.226462 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 15:07:34.226701 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 15:07:34.227936 10140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:34.356045 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 10017
I20260619 15:07:34.383984 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 9656
I20260619 15:07:34.385470 9718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:34.496529 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 9656
2026-06-19T15:07:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4754 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T15:07:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:07:34Z Disabled control of system clock
I20260619 15:07:34.576308 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39489
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:39489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:07:34.924440 10201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:34.924813 10201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:34.924949 10201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:34.932598 10201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:07:34.932755 10201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:34.932868 10201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:34.932977 10201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:07:34.933061 10201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:07:34.942729 10201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:39489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:34.944149 10201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:34.946009 10201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:34.958408 10206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:34.958482 10207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:34.958509 10209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:34.959404 10201 server_base.cc:1034] running on GCE node
I20260619 15:07:34.960332 10201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:34.961830 10201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:34.963043 10201 hybrid_clock.cc:648] HybridClock initialized: now 1781881654962992 us; error 51 us; skew 500 ppm
I20260619 15:07:34.963474 10201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:07:34.970968 10201 webserver.cc:460] Webserver started at http://127.24.103.126:37509/ using document root <none> and password file <none>
I20260619 15:07:34.971756 10201 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:34.971892 10201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:34.972258 10201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:34.975440 10201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "d3200d2df88f49838b44744a49e615c3"
format_stamp: "Formatted at 2026-06-19 15:07:34 on dist-test-slave-zpbw"
server_key: "1d62f414ecc4bc0b9279749559120da8"
server_key_iv: "a64f2259f4ebdec6bd79504fd6168892"
server_key_version: "encryptionkey@0"
I20260619 15:07:34.976289 10201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "d3200d2df88f49838b44744a49e615c3"
format_stamp: "Formatted at 2026-06-19 15:07:34 on dist-test-slave-zpbw"
server_key: "1d62f414ecc4bc0b9279749559120da8"
server_key_iv: "a64f2259f4ebdec6bd79504fd6168892"
server_key_version: "encryptionkey@0"
I20260619 15:07:34.981724 10201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:07:34.985569 10215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:34.987113 10201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:07:34.987322 10201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "d3200d2df88f49838b44744a49e615c3"
format_stamp: "Formatted at 2026-06-19 15:07:34 on dist-test-slave-zpbw"
server_key: "1d62f414ecc4bc0b9279749559120da8"
server_key_iv: "a64f2259f4ebdec6bd79504fd6168892"
server_key_version: "encryptionkey@0"
I20260619 15:07:34.987562 10201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:35.004899 10201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:35.005836 10201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:35.006165 10201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:35.026322 10201 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:39489
I20260619 15:07:35.026340 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:39489 every 8 connection(s)
I20260619 15:07:35.027878 10201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:07:35.031797 10267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:35.032060 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 10201
I20260619 15:07:35.032261 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:07:35.032738 24989 external_mini_cluster.cc:1428] Setting key 3748de3ec6ee9621b8535ebf73382782
I20260619 15:07:35.044916 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3: Bootstrap starting.
I20260619 15:07:35.047601 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:35.034692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50310 (local address 127.24.103.126:39489)
0619 15:07:35.035825 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:35.035838 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:35.035892 (+ 54us) server_negotiation.cc:407] Connection header received
0619 15:07:35.038237 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:35.038263 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:35.038909 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:35.039373 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:35.039991 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:35.041296 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:35.042814 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:35.043550 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:35.043772 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:35.045063 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:35.045094 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:35.045117 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:35.045706 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:35.045789 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:35.045812 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:35.045949 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:35.046191 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 15:07:35.046548 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":378,"threads_started":1}
I20260619 15:07:35.048676 10267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:35.049861 10267 log.cc:824] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:35.052939 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3: No bootstrap required, opened a new log
I20260619 15:07:35.060282 10267 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3200d2df88f49838b44744a49e615c3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39489 } }
I20260619 15:07:35.060737 10267 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:35.060860 10267 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3200d2df88f49838b44744a49e615c3, State: Initialized, Role: FOLLOWER
I20260619 15:07:35.061509 10267 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3200d2df88f49838b44744a49e615c3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39489 } }
I20260619 15:07:35.061769 10267 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:07:35.061918 10267 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:07:35.062103 10267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:35.064442 10267 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3200d2df88f49838b44744a49e615c3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39489 } }
I20260619 15:07:35.064985 10267 leader_election.cc:302] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3200d2df88f49838b44744a49e615c3; no voters:
I20260619 15:07:35.065582 10267 leader_election.cc:288] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:07:35.065755 10272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:35.066524 10272 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [term 1 LEADER]: Becoming Leader. State: Replica: d3200d2df88f49838b44744a49e615c3, State: Running, Role: LEADER
I20260619 15:07:35.067131 10272 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3200d2df88f49838b44744a49e615c3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39489 } }
I20260619 15:07:35.067925 10267 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:07:35.070175 10274 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3200d2df88f49838b44744a49e615c3. Latest consensus state: current_term: 1 leader_uuid: "d3200d2df88f49838b44744a49e615c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3200d2df88f49838b44744a49e615c3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39489 } } }
I20260619 15:07:35.070640 10274 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:35.070287 10273 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3200d2df88f49838b44744a49e615c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3200d2df88f49838b44744a49e615c3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39489 } } }
I20260619 15:07:35.070988 10273 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:35.074050 10278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:07:35.079748 10278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:07:35.089576 10278 catalog_manager.cc:1347] Generated new cluster ID: 0021e143be274d4fa2b05a4587e1a3ce
I20260619 15:07:35.089758 10278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:07:35.104796 10278 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:07:35.106109 10278 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:07:35.113446 10278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3: Generated new TSK 0
I20260619 15:07:35.114354 10278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:07:35.122509 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39489
--builtin_ntp_servers=127.24.103.84:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:07:35.490175 10291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:35.490487 10291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:35.490578 10291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:35.498168 10291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:35.498288 10291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:35.498396 10291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:07:35.507824 10291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:35.509167 10291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:35.511132 10291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:35.524183 10297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:35.524224 10296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:35.525386 10299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:35.526667 10291 server_base.cc:1034] running on GCE node
I20260619 15:07:35.527334 10291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:35.528319 10291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:35.529527 10291 hybrid_clock.cc:648] HybridClock initialized: now 1781881655529457 us; error 62 us; skew 500 ppm
I20260619 15:07:35.529874 10291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:07:35.532301 10291 webserver.cc:460] Webserver started at http://127.24.103.65:34347/ using document root <none> and password file <none>
I20260619 15:07:35.533049 10291 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:35.533171 10291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:35.533505 10291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:35.536617 10291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "9289a912437246a5ac30fee3ab9aae14"
format_stamp: "Formatted at 2026-06-19 15:07:35 on dist-test-slave-zpbw"
server_key: "0348fee52e788afbe0829a766cdde144"
server_key_iv: "91cc55ef4e318d99667f94f5846d0a49"
server_key_version: "encryptionkey@0"
I20260619 15:07:35.537645 10291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "9289a912437246a5ac30fee3ab9aae14"
format_stamp: "Formatted at 2026-06-19 15:07:35 on dist-test-slave-zpbw"
server_key: "0348fee52e788afbe0829a766cdde144"
server_key_iv: "91cc55ef4e318d99667f94f5846d0a49"
server_key_version: "encryptionkey@0"
I20260619 15:07:35.543058 10291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 15:07:35.546666 10305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:35.548134 10291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:07:35.548339 10291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "9289a912437246a5ac30fee3ab9aae14"
format_stamp: "Formatted at 2026-06-19 15:07:35 on dist-test-slave-zpbw"
server_key: "0348fee52e788afbe0829a766cdde144"
server_key_iv: "91cc55ef4e318d99667f94f5846d0a49"
server_key_version: "encryptionkey@0"
I20260619 15:07:35.548557 10291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:35.576339 10291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:35.577236 10291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:35.577548 10291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:35.578568 10291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:35.580183 10291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:35.580273 10291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:35.580431 10291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:35.580530 10291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:35.614883 10291 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:44269
I20260619 15:07:35.614935 10417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:44269 every 8 connection(s)
I20260619 15:07:35.616369 10291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:07:35.622831 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 10291
I20260619 15:07:35.623072 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:07:35.623531 24989 external_mini_cluster.cc:1428] Setting key 2962d4cf0452a0d1caa8b05c46f7cb6e
I20260619 15:07:35.630244 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:35.618981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:58977 (local address 127.24.103.126:39489)
0619 15:07:35.619440 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:35.619446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:35.620704 (+ 1258us) server_negotiation.cc:407] Connection header received
0619 15:07:35.622480 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:35.622487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:35.622583 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:35.622718 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:35.624020 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:35.625218 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:35.627089 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:35.627817 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:35.627962 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:35.629365 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:35.629370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:35.629374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:35.629391 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:35.629431 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:35.629443 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:35.629542 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:35.629782 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 15:07:35.629891 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":216,"threads_started":1}
I20260619 15:07:35.630704 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39489
--builtin_ntp_servers=127.24.103.84:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 15:07:35.631528 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:35.619653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39489 (local address 127.24.103.65:58977)
0619 15:07:35.620538 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:35.620574 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:35.622199 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:35.622977 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:35.622986 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:35.623125 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:35.623819 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:35.623837 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:35.625406 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:35.625411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:35.626930 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:35.626943 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:35.628033 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:35.628038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:35.628241 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:35.628997 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:07:35.629000 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:35.629054 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:07:35.629140 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:35.629543 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:35.629558 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:35.629568 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 15:07:35.629766 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 15:07:35.630197 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":263,"threads_started":1}
I20260619 15:07:35.633106 10418 heartbeater.cc:347] Connected to a master server at 127.24.103.126:39489
I20260619 15:07:35.633464 10418 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:35.634408 10418 heartbeater.cc:511] Master 127.24.103.126:39489 requested a full tablet report, sending...
I20260619 15:07:35.636745 10232 ts_manager.cc:194] Registered new tserver with Master: 9289a912437246a5ac30fee3ab9aae14 (127.24.103.65:44269)
I20260619 15:07:35.638526 10232 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.65:58977
W20260619 15:07:35.972503 10422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:35.972843 10422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:35.973008 10422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:35.980358 10422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:35.980500 10422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:35.980674 10422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:07:35.990144 10422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:35.991641 10422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:35.993525 10422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:36.007658 10428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:36.007689 10430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:36.007946 10427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:36.008653 10422 server_base.cc:1034] running on GCE node
I20260619 15:07:36.009241 10422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:36.010237 10422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:36.011463 10422 hybrid_clock.cc:648] HybridClock initialized: now 1781881656011396 us; error 61 us; skew 500 ppm
I20260619 15:07:36.011758 10422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:07:36.014154 10422 webserver.cc:460] Webserver started at http://127.24.103.66:40425/ using document root <none> and password file <none>
I20260619 15:07:36.014917 10422 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:36.015012 10422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:36.015290 10422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:36.018241 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "f8516ee8e85a49609d196e966d3f4bff"
format_stamp: "Formatted at 2026-06-19 15:07:36 on dist-test-slave-zpbw"
server_key: "92a04ccb316f3e38dc497c1709bf9cd2"
server_key_iv: "8e3fb1468c56ae715375429ecbd43abc"
server_key_version: "encryptionkey@0"
I20260619 15:07:36.019059 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "f8516ee8e85a49609d196e966d3f4bff"
format_stamp: "Formatted at 2026-06-19 15:07:36 on dist-test-slave-zpbw"
server_key: "92a04ccb316f3e38dc497c1709bf9cd2"
server_key_iv: "8e3fb1468c56ae715375429ecbd43abc"
server_key_version: "encryptionkey@0"
I20260619 15:07:36.024294 10422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 15:07:36.028132 10436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:36.029681 10422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:07:36.029886 10422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "f8516ee8e85a49609d196e966d3f4bff"
format_stamp: "Formatted at 2026-06-19 15:07:36 on dist-test-slave-zpbw"
server_key: "92a04ccb316f3e38dc497c1709bf9cd2"
server_key_iv: "8e3fb1468c56ae715375429ecbd43abc"
server_key_version: "encryptionkey@0"
I20260619 15:07:36.030126 10422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:36.049553 10422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:36.050454 10422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:36.050890 10422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:36.051934 10422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:36.053705 10422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:36.053840 10422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:36.053982 10422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:36.054075 10422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:36.088354 10422 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:38151
I20260619 15:07:36.088413 10548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:38151 every 8 connection(s)
I20260619 15:07:36.089736 10422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:07:36.099512 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 10422
I20260619 15:07:36.099740 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:07:36.100200 24989 external_mini_cluster.cc:1428] Setting key b88a66e11b451412f663563d2395b6f8
I20260619 15:07:36.102504 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:36.092436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:37201 (local address 127.24.103.126:39489)
0619 15:07:36.092611 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:36.092616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:36.094001 (+ 1385us) server_negotiation.cc:407] Connection header received
0619 15:07:36.096042 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:36.096048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:36.096147 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:36.096257 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:36.097350 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.098067 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.099717 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.100164 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.100300 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:36.101696 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:36.101701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:36.101705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:36.101721 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:36.101752 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:36.101763 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:36.101855 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:36.102091 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 15:07:36.102189 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 15:07:36.103185 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39489
--builtin_ntp_servers=127.24.103.84:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 15:07:36.103746 10551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:36.092948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39489 (local address 127.24.103.66:37201)
0619 15:07:36.093843 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:36.093879 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:36.095739 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:36.096490 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:36.096499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:36.096588 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:36.097169 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:36.097185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.098214 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.098222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:36.099575 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:36.099587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.100388 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.100394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:36.100617 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:36.101348 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:07:36.101351 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:36.101399 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:07:36.101485 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:36.101855 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:36.101868 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:36.101878 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 15:07:36.102082 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 15:07:36.102499 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":276,"threads_started":1}
I20260619 15:07:36.105199 10549 heartbeater.cc:347] Connected to a master server at 127.24.103.126:39489
I20260619 15:07:36.105574 10549 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:36.106518 10549 heartbeater.cc:511] Master 127.24.103.126:39489 requested a full tablet report, sending...
I20260619 15:07:36.108201 10232 ts_manager.cc:194] Registered new tserver with Master: f8516ee8e85a49609d196e966d3f4bff (127.24.103.66:38151)
I20260619 15:07:36.109072 10232 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.66:37201
W20260619 15:07:36.441263 10552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:36.441557 10552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:36.441648 10552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:36.448844 10552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:36.448957 10552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:36.449066 10552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:07:36.462452 10552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:36.463852 10552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:36.465670 10552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:36.479951 10557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:36.479970 10558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:36.484050 10560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:36.484838 10552 server_base.cc:1034] running on GCE node
I20260619 15:07:36.485790 10552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:36.487254 10552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:36.488571 10552 hybrid_clock.cc:648] HybridClock initialized: now 1781881656488490 us; error 59 us; skew 500 ppm
I20260619 15:07:36.489089 10552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:07:36.492525 10552 webserver.cc:460] Webserver started at http://127.24.103.67:43633/ using document root <none> and password file <none>
I20260619 15:07:36.493645 10552 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:36.493795 10552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:36.494225 10552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:36.499102 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "f8f1d9642efd4a9da75d35719b010388"
format_stamp: "Formatted at 2026-06-19 15:07:36 on dist-test-slave-zpbw"
server_key: "393bc3b55b866413a7a43aa05e9e54cf"
server_key_iv: "2b14d805b8c8f750e5fbd40cb321e404"
server_key_version: "encryptionkey@0"
I20260619 15:07:36.500357 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "f8f1d9642efd4a9da75d35719b010388"
format_stamp: "Formatted at 2026-06-19 15:07:36 on dist-test-slave-zpbw"
server_key: "393bc3b55b866413a7a43aa05e9e54cf"
server_key_iv: "2b14d805b8c8f750e5fbd40cb321e404"
server_key_version: "encryptionkey@0"
I20260619 15:07:36.508515 10552 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260619 15:07:36.514228 10566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:36.516212 10552 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 15:07:36.516439 10552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "f8f1d9642efd4a9da75d35719b010388"
format_stamp: "Formatted at 2026-06-19 15:07:36 on dist-test-slave-zpbw"
server_key: "393bc3b55b866413a7a43aa05e9e54cf"
server_key_iv: "2b14d805b8c8f750e5fbd40cb321e404"
server_key_version: "encryptionkey@0"
I20260619 15:07:36.516754 10552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:36.587411 10552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:36.588402 10552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:36.588732 10552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:36.589882 10552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:36.591521 10552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:36.591609 10552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:36.591747 10552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:36.591853 10552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:36.634755 10552 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:34109
I20260619 15:07:36.634830 10678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:34109 every 8 connection(s)
I20260619 15:07:36.636237 10552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:07:36.641983 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 10552
I20260619 15:07:36.642179 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:07:36.642660 10418 heartbeater.cc:503] Master 127.24.103.126:39489 was elected leader, sending a full tablet report...
I20260619 15:07:36.642745 24989 external_mini_cluster.cc:1428] Setting key 1311e99f71ac4e398d8e108a74b47ee5
I20260619 15:07:36.651052 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:36.639006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:54901 (local address 127.24.103.126:39489)
0619 15:07:36.639594 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:36.639602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:36.640938 (+ 1336us) server_negotiation.cc:407] Connection header received
0619 15:07:36.643577 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:36.643583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:36.643671 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:36.643778 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:36.645367 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.646374 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.648050 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.648490 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.648628 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:36.650089 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:36.650094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:36.650097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:36.650117 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:36.650161 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:36.650172 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:36.650269 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:36.650502 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 15:07:36.650605 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":290,"threads_started":1}
I20260619 15:07:36.652148 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:36.639676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39489 (local address 127.24.103.67:54901)
0619 15:07:36.640749 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:36.640805 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:36.643122 (+ 2317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:36.644222 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:36.644237 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:36.644370 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:36.645192 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:36.645210 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.646526 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.646532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:36.647902 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:36.647915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.648721 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.648728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:36.648976 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:36.649731 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:07:36.649733 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:36.649775 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:07:36.649873 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:36.650269 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:36.650282 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:36.650293 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 15:07:36.650490 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 15:07:36.650990 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":336,"threads_started":1}
I20260619 15:07:36.653623 10679 heartbeater.cc:347] Connected to a master server at 127.24.103.126:39489
I20260619 15:07:36.653967 10679 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:36.654743 10679 heartbeater.cc:511] Master 127.24.103.126:39489 requested a full tablet report, sending...
I20260619 15:07:36.656409 10232 ts_manager.cc:194] Registered new tserver with Master: f8f1d9642efd4a9da75d35719b010388 (127.24.103.67:34109)
I20260619 15:07:36.657877 10232 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.67:54901
I20260619 15:07:36.660789 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:07:36.672827 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:36.665316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.24.103.126:39489)
0619 15:07:36.665569 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:36.665578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:36.665958 (+ 380us) server_negotiation.cc:407] Connection header received
0619 15:07:36.666117 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:36.666125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:36.666228 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:36.666377 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:36.666853 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.668233 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.669783 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.670627 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.670815 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:36.671898 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:36.671908 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:36.671915 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:36.671945 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:36.671992 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:36.672011 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:36.672122 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:36.672210 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 15:07:36.672359 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:07:36.677570 10232 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:07:36.685065 10232 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:07:36.727507 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:36.699280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:34109 (local address 127.0.0.1:43054)
0619 15:07:36.715286 (+ 16006us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:36.715303 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:36.715432 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:36.716362 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:36.716369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:36.716400 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:36.717403 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:36.717416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.721580 (+ 4164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.721585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:36.724467 (+ 2882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:36.724479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.726463 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.726473 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:36.726671 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:36.726742 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 15:07:36.726886 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 15:07:36.727083 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15884,"mutex_wait_us":216,"thread_start_us":16169,"threads_started":1}
I20260619 15:07:36.728226 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:36.698618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.24.103.65:44269)
0619 15:07:36.713935 (+ 15317us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:36.713947 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:36.716656 (+ 2709us) server_negotiation.cc:407] Connection header received
0619 15:07:36.716792 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:36.716804 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:36.717201 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:36.717436 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:36.718633 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.721234 (+ 2601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.724674 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.726218 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.726368 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:36.726670 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:36.727674 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0619 15:07:36.727866 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15093,"thread_start_us":15332,"threads_started":1}
I20260619 15:07:36.727553 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:36.698616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44269 (local address 127.0.0.1:40586)
0619 15:07:36.714899 (+ 16283us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:36.714952 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:36.715182 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:36.717808 (+ 2626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:36.717817 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:36.717873 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:36.718381 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:36.718397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.721382 (+ 2985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.721389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:36.724414 (+ 3025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:36.724434 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.726829 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.726835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:36.727000 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:36.727029 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 15:07:36.727106 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 15:07:36.727219 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15898,"mutex_wait_us":167,"thread_start_us":15712,"threads_started":1}
I20260619 15:07:36.729593 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:36.699588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.24.103.67:34109)
0619 15:07:36.712890 (+ 13302us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:36.712904 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:36.715602 (+ 2698us) server_negotiation.cc:407] Connection header received
0619 15:07:36.715742 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:36.715754 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:36.716108 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:36.716652 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:36.719925 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.721423 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.724674 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.726218 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.728280 (+ 2062us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:36.728835 (+ 555us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:36.728914 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 15:07:36.729228 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13015,"thread_start_us":13321,"threads_started":1}
I20260619 15:07:36.733089 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:36.715282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:38151 (local address 127.0.0.1:47016)
0619 15:07:36.719601 (+ 4319us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:36.719619 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:36.719743 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:36.720631 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:36.720639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:36.720672 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:36.721129 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:36.721143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.724677 (+ 3534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.724685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:36.727365 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:36.727385 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.732031 (+ 4646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.732040 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:36.732309 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:36.732360 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 15:07:36.732456 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 15:07:36.732641 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4196,"thread_start_us":4000,"threads_started":1}
I20260619 15:07:36.732772 10353 tablet_service.cc:1467] Processing CreateTablet for tablet bebf3bc2ce694a699b6277bbcfc9cfac (DEFAULT_TABLE table=test-table [id=63f05a84b83c4a328448a9ffa0b2dc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:36.733220 10614 tablet_service.cc:1467] Processing CreateTablet for tablet bebf3bc2ce694a699b6277bbcfc9cfac (DEFAULT_TABLE table=test-table [id=63f05a84b83c4a328448a9ffa0b2dc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:36.734248 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:36.715529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.24.103.66:38151)
0619 15:07:36.719214 (+ 3685us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:36.719227 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:36.719747 (+ 520us) server_negotiation.cc:407] Connection header received
0619 15:07:36.719871 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:36.719881 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:36.720152 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:36.720364 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:36.721359 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.724488 (+ 3129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.727788 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:36.731455 (+ 3667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:36.731603 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:36.731887 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:36.733367 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0619 15:07:36.733756 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3533,"thread_start_us":3414,"threads_started":1}
I20260619 15:07:36.734444 10353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebf3bc2ce694a699b6277bbcfc9cfac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:36.735145 10614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebf3bc2ce694a699b6277bbcfc9cfac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:36.744925 10484 tablet_service.cc:1467] Processing CreateTablet for tablet bebf3bc2ce694a699b6277bbcfc9cfac (DEFAULT_TABLE table=test-table [id=63f05a84b83c4a328448a9ffa0b2dc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:36.746856 10484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebf3bc2ce694a699b6277bbcfc9cfac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:36.758668 10698 tablet_bootstrap.cc:492] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14: Bootstrap starting.
I20260619 15:07:36.761832 10698 tablet_bootstrap.cc:654] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:36.762575 10699 tablet_bootstrap.cc:492] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388: Bootstrap starting.
I20260619 15:07:36.763242 10698 log.cc:824] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:36.766575 10701 tablet_bootstrap.cc:492] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff: Bootstrap starting.
I20260619 15:07:36.768136 10699 tablet_bootstrap.cc:654] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:36.769460 10699 log.cc:824] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:36.769935 10698 tablet_bootstrap.cc:492] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14: No bootstrap required, opened a new log
I20260619 15:07:36.769991 10701 tablet_bootstrap.cc:654] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:36.770999 10698 ts_tablet_manager.cc:1397] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260619 15:07:36.771059 10701 log.cc:824] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:36.773605 10699 tablet_bootstrap.cc:492] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388: No bootstrap required, opened a new log
I20260619 15:07:36.773909 10699 ts_tablet_manager.cc:1397] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20260619 15:07:36.774106 10701 tablet_bootstrap.cc:492] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff: No bootstrap required, opened a new log
I20260619 15:07:36.774379 10701 ts_tablet_manager.cc:1397] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 15:07:36.778721 10698 raft_consensus.cc:348] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9289a912437246a5ac30fee3ab9aae14" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44269 } } peers { permanent_uuid: "f8516ee8e85a49609d196e966d3f4bff" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38151 } } peers { permanent_uuid: "f8f1d9642efd4a9da75d35719b010388" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34109 } }
I20260619 15:07:36.779225 10698 raft_consensus.cc:374] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:36.779323 10698 raft_consensus.cc:729] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9289a912437246a5ac30fee3ab9aae14, State: Initialized, Role: FOLLOWER
I20260619 15:07:36.780215 10698 consensus_queue.cc:260] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9289a912437246a5ac30fee3ab9aae14" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44269 } } peers { permanent_uuid: "f8516ee8e85a49609d196e966d3f4bff" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38151 } } peers { permanent_uuid: "f8f1d9642efd4a9da75d35719b010388" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34109 } }
I20260619 15:07:36.782605 10698 ts_tablet_manager.cc:1428] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260619 15:07:36.785101 10701 raft_consensus.cc:348] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9289a912437246a5ac30fee3ab9aae14" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44269 } } peers { permanent_uuid: "f8516ee8e85a49609d196e966d3f4bff" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38151 } } peers { permanent_uuid: "f8f1d9642efd4a9da75d35719b010388" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34109 } }
I20260619 15:07:36.785101 10699 raft_consensus.cc:348] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9289a912437246a5ac30fee3ab9aae14" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44269 } } peers { permanent_uuid: "f8516ee8e85a49609d196e966d3f4bff" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38151 } } peers { permanent_uuid: "f8f1d9642efd4a9da75d35719b010388" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34109 } }
I20260619 15:07:36.785583 10701 raft_consensus.cc:374] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:36.785583 10699 raft_consensus.cc:374] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:36.785689 10699 raft_consensus.cc:729] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f1d9642efd4a9da75d35719b010388, State: Initialized, Role: FOLLOWER
I20260619 15:07:36.785689 10701 raft_consensus.cc:729] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8516ee8e85a49609d196e966d3f4bff, State: Initialized, Role: FOLLOWER
I20260619 15:07:36.786332 10699 consensus_queue.cc:260] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9289a912437246a5ac30fee3ab9aae14" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44269 } } peers { permanent_uuid: "f8516ee8e85a49609d196e966d3f4bff" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38151 } } peers { permanent_uuid: "f8f1d9642efd4a9da75d35719b010388" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34109 } }
I20260619 15:07:36.786342 10701 consensus_queue.cc:260] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9289a912437246a5ac30fee3ab9aae14" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44269 } } peers { permanent_uuid: "f8516ee8e85a49609d196e966d3f4bff" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38151 } } peers { permanent_uuid: "f8f1d9642efd4a9da75d35719b010388" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34109 } }
I20260619 15:07:36.787535 10549 heartbeater.cc:503] Master 127.24.103.126:39489 was elected leader, sending a full tablet report...
I20260619 15:07:36.788584 10701 ts_tablet_manager.cc:1428] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260619 15:07:36.789090 10679 heartbeater.cc:503] Master 127.24.103.126:39489 was elected leader, sending a full tablet report...
I20260619 15:07:36.789631 10699 ts_tablet_manager.cc:1428] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388: Time spent starting tablet: real 0.016s user 0.011s sys 0.006s
W20260619 15:07:36.841869 10550 tablet.cc:2369] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:07:36.868911 10419 tablet.cc:2369] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:07:36.888227 10680 tablet.cc:2369] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:07:37.153177 10704 raft_consensus.cc:482] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:07:37.153537 10704 raft_consensus.cc:504] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9289a912437246a5ac30fee3ab9aae14" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44269 } } peers { permanent_uuid: "f8516ee8e85a49609d196e966d3f4bff" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38151 } } peers { permanent_uuid: "f8f1d9642efd4a9da75d35719b010388" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34109 } }
I20260619 15:07:37.155531 10704 leader_election.cc:288] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8516ee8e85a49609d196e966d3f4bff (127.24.103.66:38151), f8f1d9642efd4a9da75d35719b010388 (127.24.103.67:34109)
I20260619 15:07:37.164297 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.156034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:33181 (local address 127.24.103.66:38151)
0619 15:07:37.156374 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:37.156382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:37.157402 (+ 1020us) server_negotiation.cc:407] Connection header received
0619 15:07:37.157578 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:37.157582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:37.157643 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:37.157722 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:37.158315 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.159339 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.161926 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.163198 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.163357 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:37.163430 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:37.163905 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 15:07:37.164035 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260619 15:07:37.164300 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.156434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:43947 (local address 127.24.103.67:34109)
0619 15:07:37.156777 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:37.156785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:37.156871 (+ 86us) server_negotiation.cc:407] Connection header received
0619 15:07:37.157173 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:37.157178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:37.157237 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:37.157318 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:37.157999 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.159081 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.161927 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.163198 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.163371 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:37.163430 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:37.163905 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 15:07:37.164034 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260619 15:07:37.164486 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.156473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:38151 (local address 127.24.103.65:33181)
0619 15:07:37.157259 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:37.157271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:37.157414 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:37.157786 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:37.157800 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:37.157830 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:37.158160 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:37.158169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.159463 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.159468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:37.161668 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:37.161684 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.163436 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.163444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:37.163692 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:37.163757 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 15:07:37.163859 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 15:07:37.164020 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":100,"thread_start_us":456,"threads_started":1}
I20260619 15:07:37.164824 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.155978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:34109 (local address 127.24.103.65:43947)
0619 15:07:37.156749 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:37.156770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:37.157008 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:37.157380 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:37.157384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:37.157472 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:37.157834 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:37.157846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.159204 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.159208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:37.161670 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:37.161684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.163453 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.163460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:37.163686 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:37.163757 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 15:07:37.163859 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 15:07:37.164020 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":178,"spinlock_wait_cycles":27392,"thread_start_us":411,"threads_started":1}
I20260619 15:07:37.165678 10504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf3bc2ce694a699b6277bbcfc9cfac" candidate_uuid: "9289a912437246a5ac30fee3ab9aae14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8516ee8e85a49609d196e966d3f4bff" is_pre_election: true
I20260619 15:07:37.165807 10634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf3bc2ce694a699b6277bbcfc9cfac" candidate_uuid: "9289a912437246a5ac30fee3ab9aae14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f1d9642efd4a9da75d35719b010388" is_pre_election: true
I20260619 15:07:37.166229 10634 raft_consensus.cc:2454] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9289a912437246a5ac30fee3ab9aae14 in term 0.
I20260619 15:07:37.166208 10504 raft_consensus.cc:2454] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9289a912437246a5ac30fee3ab9aae14 in term 0.
I20260619 15:07:37.166955 10308 leader_election.cc:302] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9289a912437246a5ac30fee3ab9aae14, f8f1d9642efd4a9da75d35719b010388; no voters:
I20260619 15:07:37.167403 10704 raft_consensus.cc:2781] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:07:37.167585 10704 raft_consensus.cc:482] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:07:37.167721 10704 raft_consensus.cc:3037] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:37.169886 10704 raft_consensus.cc:504] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9289a912437246a5ac30fee3ab9aae14" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44269 } } peers { permanent_uuid: "f8516ee8e85a49609d196e966d3f4bff" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38151 } } peers { permanent_uuid: "f8f1d9642efd4a9da75d35719b010388" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34109 } }
I20260619 15:07:37.170531 10704 leader_election.cc:288] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [CANDIDATE]: Term 1 election: Requested vote from peers f8516ee8e85a49609d196e966d3f4bff (127.24.103.66:38151), f8f1d9642efd4a9da75d35719b010388 (127.24.103.67:34109)
I20260619 15:07:37.171000 10504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf3bc2ce694a699b6277bbcfc9cfac" candidate_uuid: "9289a912437246a5ac30fee3ab9aae14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8516ee8e85a49609d196e966d3f4bff"
I20260619 15:07:37.171078 10634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf3bc2ce694a699b6277bbcfc9cfac" candidate_uuid: "9289a912437246a5ac30fee3ab9aae14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f1d9642efd4a9da75d35719b010388"
I20260619 15:07:37.171204 10504 raft_consensus.cc:3037] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:37.171348 10634 raft_consensus.cc:3037] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:37.173758 10504 raft_consensus.cc:2454] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9289a912437246a5ac30fee3ab9aae14 in term 1.
I20260619 15:07:37.173794 10634 raft_consensus.cc:2454] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9289a912437246a5ac30fee3ab9aae14 in term 1.
I20260619 15:07:37.174209 10309 leader_election.cc:302] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9289a912437246a5ac30fee3ab9aae14, f8516ee8e85a49609d196e966d3f4bff; no voters:
I20260619 15:07:37.174590 10704 raft_consensus.cc:2781] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:37.175108 10704 raft_consensus.cc:686] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 1 LEADER]: Becoming Leader. State: Replica: 9289a912437246a5ac30fee3ab9aae14, State: Running, Role: LEADER
I20260619 15:07:37.175654 10704 consensus_queue.cc:237] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9289a912437246a5ac30fee3ab9aae14" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44269 } } peers { permanent_uuid: "f8516ee8e85a49609d196e966d3f4bff" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38151 } } peers { permanent_uuid: "f8f1d9642efd4a9da75d35719b010388" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34109 } }
I20260619 15:07:37.180130 10231 catalog_manager.cc:5495] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9289a912437246a5ac30fee3ab9aae14 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "9289a912437246a5ac30fee3ab9aae14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9289a912437246a5ac30fee3ab9aae14" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8516ee8e85a49609d196e966d3f4bff" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8f1d9642efd4a9da75d35719b010388" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34109 } health_report { overall_health: UNKNOWN } } }
I20260619 15:07:37.278504 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.272336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.24.103.65:44269)
0619 15:07:37.273041 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:37.273049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:37.273074 (+ 25us) server_negotiation.cc:407] Connection header received
0619 15:07:37.273332 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:37.273336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:37.273423 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:37.273589 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:07:37.274082 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.274899 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.276305 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.276712 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.276805 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:37.277281 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:07:37.277858 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:07:37.277978 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:37.278090 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 15:07:37.278162 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":456,"threads_started":1}
I20260619 15:07:37.286646 10504 raft_consensus.cc:1261] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff [term 1 FOLLOWER]: Refusing update from remote peer 9289a912437246a5ac30fee3ab9aae14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:07:37.286644 10634 raft_consensus.cc:1261] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388 [term 1 FOLLOWER]: Refusing update from remote peer 9289a912437246a5ac30fee3ab9aae14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:07:37.287680 10709 consensus_queue.cc:1035] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8516ee8e85a49609d196e966d3f4bff" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:07:37.287999 10704 consensus_queue.cc:1035] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f1d9642efd4a9da75d35719b010388" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:07:37.302950 10714 mvcc.cc:205] Tried to move back new op lower bound from 7298587268236312576 to 7298587267795386368. Current Snapshot: MvccSnapshot[applied={T|T < 7298587268236312576}]
I20260619 15:07:37.319914 10232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:07:37.320171 10232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:07:37.324177 10232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3200d2df88f49838b44744a49e615c3: Sending DeleteTablet for 3 replicas of tablet bebf3bc2ce694a699b6277bbcfc9cfac
I20260619 15:07:37.325517 10614 tablet_service.cc:1514] Processing DeleteTablet for tablet bebf3bc2ce694a699b6277bbcfc9cfac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:37 UTC) from {username='slave'} at 127.0.0.1:43054
I20260619 15:07:37.325737 10353 tablet_service.cc:1514] Processing DeleteTablet for tablet bebf3bc2ce694a699b6277bbcfc9cfac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:37 UTC) from {username='slave'} at 127.0.0.1:40586
I20260619 15:07:37.326256 10484 tablet_service.cc:1514] Processing DeleteTablet for tablet bebf3bc2ce694a699b6277bbcfc9cfac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:37 UTC) from {username='slave'} at 127.0.0.1:47016
I20260619 15:07:37.327085 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 10291
I20260619 15:07:37.338418 10724 tablet_replica.cc:330] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388: stopping tablet replica
I20260619 15:07:37.339025 10724 raft_consensus.cc:2229] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:07:37.339600 10724 raft_consensus.cc:2258] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:37.340466 10725 tablet_replica.cc:330] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14: stopping tablet replica
I20260619 15:07:37.340914 10725 raft_consensus.cc:2229] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:07:37.341661 10725 raft_consensus.cc:2258] T bebf3bc2ce694a699b6277bbcfc9cfac P 9289a912437246a5ac30fee3ab9aae14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:37.342463 10726 tablet_replica.cc:330] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff: stopping tablet replica
I20260619 15:07:37.342885 10726 raft_consensus.cc:2229] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:07:37.343394 10726 raft_consensus.cc:2258] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:37.345289 10726 ts_tablet_manager.cc:1905] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:37.348501 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.328687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.24.103.65:44269)
0619 15:07:37.328870 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:37.328877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:37.339664 (+ 10787us) server_negotiation.cc:407] Connection header received
0619 15:07:37.340106 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:37.340115 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:37.340247 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:37.340332 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:37.341290 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.342488 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.344593 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.345267 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.345492 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:37.346742 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:37.346753 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:37.346771 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:37.347606 (+ 835us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:37.347686 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:37.347712 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:37.347823 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:37.348011 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 15:07:37.348118 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 15:07:37.349412 10413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:37.351687 10724 ts_tablet_manager.cc:1905] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:37.352010 10726 ts_tablet_manager.cc:1918] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:37.352185 10726 log.cc:1196] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/bebf3bc2ce694a699b6277bbcfc9cfac
I20260619 15:07:37.352619 10726 ts_tablet_manager.cc:1939] T bebf3bc2ce694a699b6277bbcfc9cfac P f8516ee8e85a49609d196e966d3f4bff: Deleting consensus metadata
I20260619 15:07:37.355054 10219 catalog_manager.cc:4841] TS f8516ee8e85a49609d196e966d3f4bff (127.24.103.66:38151): tablet bebf3bc2ce694a699b6277bbcfc9cfac (table test-table [id=63f05a84b83c4a328448a9ffa0b2dc2e]) successfully deleted
I20260619 15:07:37.358210 10724 ts_tablet_manager.cc:1918] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:37.358354 10724 log.cc:1196] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/bebf3bc2ce694a699b6277bbcfc9cfac
I20260619 15:07:37.358872 10724 ts_tablet_manager.cc:1939] T bebf3bc2ce694a699b6277bbcfc9cfac P f8f1d9642efd4a9da75d35719b010388: Deleting consensus metadata
I20260619 15:07:37.360498 10218 catalog_manager.cc:4841] TS f8f1d9642efd4a9da75d35719b010388 (127.24.103.67:34109): tablet bebf3bc2ce694a699b6277bbcfc9cfac (table test-table [id=63f05a84b83c4a328448a9ffa0b2dc2e]) successfully deleted
I20260619 15:07:37.448436 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 10291
I20260619 15:07:37.467104 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 10422
W20260619 15:07:37.467309 10219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.103.65:44269 (error 108)
I20260619 15:07:37.470001 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.469062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44269 (local address 127.0.0.1:40596)
0619 15:07:37.469619 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:37.469787 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":241,"threads_started":1}
W20260619 15:07:37.470283 10219 catalog_manager.cc:4584] TS 9289a912437246a5ac30fee3ab9aae14 (127.24.103.65:44269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bebf3bc2ce694a699b6277bbcfc9cfac: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44269: connect: Connection refused (error 111)
I20260619 15:07:37.474998 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.467942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.24.103.66:38151)
0619 15:07:37.468154 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:37.468161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:37.468340 (+ 179us) server_negotiation.cc:407] Connection header received
0619 15:07:37.468536 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:37.468544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:37.468654 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:37.468777 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:37.469186 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.470293 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.471739 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.472174 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.472327 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:37.473394 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:37.473403 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:37.473417 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:37.474008 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:37.474057 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:37.474076 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:37.474230 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:37.474346 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 15:07:37.474452 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 15:07:37.475677 10544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:37.524083 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.523532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44269 (local address 127.0.0.1:40598)
0619 15:07:37.523786 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:37.523903 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 15:07:37.525285 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.524808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44269 (local address 127.0.0.1:40614)
0619 15:07:37.524978 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:37.525083 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 15:07:37.566020 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 10422
I20260619 15:07:37.583359 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 10552
I20260619 15:07:37.586338 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.585784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44269 (local address 127.0.0.1:40630)
0619 15:07:37.586028 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:37.586142 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 15:07:37.587525 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.587063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44269 (local address 127.0.0.1:40646)
0619 15:07:37.587244 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:37.587346 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 15:07:37.591373 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.584210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.24.103.67:34109)
0619 15:07:37.584436 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:37.584441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:37.584463 (+ 22us) server_negotiation.cc:407] Connection header received
0619 15:07:37.584658 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:37.584662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:37.584761 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:37.584879 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:37.585310 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.586434 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.588189 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:37.588618 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:37.588762 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:37.589856 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:37.589865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:37.589880 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:37.590480 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:37.590530 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:37.590550 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:37.590699 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:37.590875 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 15:07:37.590997 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:07:37.592092 10674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:37.668497 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.667914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44269 (local address 127.0.0.1:40658)
0619 15:07:37.668184 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:37.668299 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 15:07:37.669751 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:37.669236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44269 (local address 127.0.0.1:40664)
0619 15:07:37.669485 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:37.669577 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 15:07:37.680961 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 10552
I20260619 15:07:37.697703 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 10201
I20260619 15:07:37.698936 10262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:37.809921 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 10201
2026-06-19T15:07:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3313 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:07:37 dist-test-slave-zpbw krb5kdc[10736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:07:37 dist-test-slave-zpbw krb5kdc[10736](info): set up 2 sockets
Jun 19 15:07:37 dist-test-slave-zpbw krb5kdc[10736](info): commencing operation
krb5kdc: starting...
W20260619 15:07:38.927248 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:07:39 dist-test-slave-zpbw krb5kdc[10736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:07:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:07:39.181944 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:46791
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:39.516584 10749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:39.516981 10749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:39.517158 10749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:39.524667 10749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:07:39.524847 10749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:39.524969 10749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:39.525069 10749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:07:39.525166 10749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:07:39.535368 10749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:46791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:39.536948 10749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:39.538393 10749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:39.551061 10757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:39.550949 10754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:39.550971 10755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:39.551852 10749 server_base.cc:1034] running on GCE node
I20260619 15:07:39.552706 10749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:39.554360 10749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:39.555620 10749 hybrid_clock.cc:648] HybridClock initialized: now 1781881659555574 us; error 36 us; skew 500 ppm
I20260619 15:07:39.555989 10749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:39 dist-test-slave-zpbw krb5kdc[10736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881659, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:39.559273 10749 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:07:39.560982 10749 webserver.cc:460] Webserver started at http://127.24.103.126:34957/ using document root <none> and password file <none>
I20260619 15:07:39.561689 10749 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:39.561797 10749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:39.562064 10749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:39.565250 10749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "9885e61d825b4db49b446d48dc1d3f8a"
format_stamp: "Formatted at 2026-06-19 15:07:39 on dist-test-slave-zpbw"
server_key: "13c357de3bf171a3387d484bfde862d1"
server_key_iv: "d6a318cc3dcade5a6d011b706a8509bc"
server_key_version: "encryptionkey@0"
I20260619 15:07:39.565992 10749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "9885e61d825b4db49b446d48dc1d3f8a"
format_stamp: "Formatted at 2026-06-19 15:07:39 on dist-test-slave-zpbw"
server_key: "13c357de3bf171a3387d484bfde862d1"
server_key_iv: "d6a318cc3dcade5a6d011b706a8509bc"
server_key_version: "encryptionkey@0"
I20260619 15:07:39.571513 10749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:07:39.575146 10764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:39.576606 10749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 15:07:39.576812 10749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "9885e61d825b4db49b446d48dc1d3f8a"
format_stamp: "Formatted at 2026-06-19 15:07:39 on dist-test-slave-zpbw"
server_key: "13c357de3bf171a3387d484bfde862d1"
server_key_iv: "d6a318cc3dcade5a6d011b706a8509bc"
server_key_version: "encryptionkey@0"
I20260619 15:07:39.577019 10749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:39.606509 10749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:39.611361 10749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:39.611693 10749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:39.630884 10749 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:46791
I20260619 15:07:39.630869 10815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:46791 every 8 connection(s)
I20260619 15:07:39.632398 10749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:07:39.636448 10816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:39.636907 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 10749
I20260619 15:07:39.637102 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:07:39.637691 24989 external_mini_cluster.cc:1428] Setting key 39e97df411db5b8912576261d7c248fb
I20260619 15:07:39.651284 10816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a: Bootstrap starting.
Jun 19 15:07:39 dist-test-slave-zpbw krb5kdc[10736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:07:39.655210 10816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:39.656277 10816 log.cc:824] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:39.659251 10816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a: No bootstrap required, opened a new log
I20260619 15:07:39.663348 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:39.639762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.24.103.126:46791)
0619 15:07:39.640625 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:39.640638 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:39.640682 (+ 44us) server_negotiation.cc:407] Connection header received
0619 15:07:39.643082 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:39.643108 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:39.643548 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:39.643904 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:39.645760 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:39.647145 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:39.649117 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:39.649832 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:39.649964 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:39.655138 (+ 5174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:39.655148 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:39.655164 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:39.655212 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:39.658428 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:39.659243 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:39.659250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:39.659256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:39.659502 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:39.660070 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:39.660076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:39.660081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:39.660868 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:39.661092 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:39.661719 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 15:07:39.662183 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":419,"threads_started":1}
I20260619 15:07:39.668697 10816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9885e61d825b4db49b446d48dc1d3f8a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46791 } }
I20260619 15:07:39.669121 10816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:39.669237 10816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9885e61d825b4db49b446d48dc1d3f8a, State: Initialized, Role: FOLLOWER
I20260619 15:07:39.669801 10816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9885e61d825b4db49b446d48dc1d3f8a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46791 } }
I20260619 15:07:39.670046 10816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:07:39.670189 10816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:07:39.670358 10816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:39.672920 10816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9885e61d825b4db49b446d48dc1d3f8a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46791 } }
I20260619 15:07:39.673451 10816 leader_election.cc:302] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9885e61d825b4db49b446d48dc1d3f8a; no voters:
I20260619 15:07:39.674091 10816 leader_election.cc:288] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:07:39.674237 10821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:39.675010 10821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [term 1 LEADER]: Becoming Leader. State: Replica: 9885e61d825b4db49b446d48dc1d3f8a, State: Running, Role: LEADER
I20260619 15:07:39.675592 10821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9885e61d825b4db49b446d48dc1d3f8a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46791 } }
I20260619 15:07:39.676376 10816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:07:39.679023 10822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9885e61d825b4db49b446d48dc1d3f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9885e61d825b4db49b446d48dc1d3f8a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46791 } } }
I20260619 15:07:39.679514 10822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:39.678999 10823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9885e61d825b4db49b446d48dc1d3f8a. Latest consensus state: current_term: 1 leader_uuid: "9885e61d825b4db49b446d48dc1d3f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9885e61d825b4db49b446d48dc1d3f8a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46791 } } }
I20260619 15:07:39.679862 10823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:39.680595 10828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:07:39.686640 10828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:07:39.696213 10828 catalog_manager.cc:1347] Generated new cluster ID: d7d221bd40694e178917fcd53b3aa3f9
I20260619 15:07:39.696357 10828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:07:39.709259 10828 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:07:39.711340 10828 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:07:39.723948 10828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a: Generated new TSK 0
I20260619 15:07:39.724872 10828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:07:39.792448 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:46791
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:40.152554 10842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:40.152887 10842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:40.152985 10842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:40.160667 10842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:40.160791 10842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:40.160966 10842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:07:40.170979 10842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:46791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:40.172341 10842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:40.173682 10842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:40.188354 10847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:40.188396 10848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:40.188582 10850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:40.189110 10842 server_base.cc:1034] running on GCE node
I20260619 15:07:40.189761 10842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:40.190901 10842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:40.192122 10842 hybrid_clock.cc:648] HybridClock initialized: now 1781881660192049 us; error 64 us; skew 500 ppm
I20260619 15:07:40.192435 10842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:40 dist-test-slave-zpbw krb5kdc[10736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881660, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:40.200069 10842 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:07:40.201772 10842 webserver.cc:460] Webserver started at http://127.24.103.65:35307/ using document root <none> and password file <none>
I20260619 15:07:40.202539 10842 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:40.202684 10842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:40.203063 10842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:40.206130 10842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "8ffffd945938474c810873e349c1112a"
format_stamp: "Formatted at 2026-06-19 15:07:40 on dist-test-slave-zpbw"
server_key: "4d2e964759ee9075eed277c7d0e195bd"
server_key_iv: "bb9ad9792770e565aad6f0c49d2e9f33"
server_key_version: "encryptionkey@0"
I20260619 15:07:40.207041 10842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "8ffffd945938474c810873e349c1112a"
format_stamp: "Formatted at 2026-06-19 15:07:40 on dist-test-slave-zpbw"
server_key: "4d2e964759ee9075eed277c7d0e195bd"
server_key_iv: "bb9ad9792770e565aad6f0c49d2e9f33"
server_key_version: "encryptionkey@0"
I20260619 15:07:40.212468 10842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:07:40.216104 10857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:40.217463 10842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:07:40.217648 10842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "8ffffd945938474c810873e349c1112a"
format_stamp: "Formatted at 2026-06-19 15:07:40 on dist-test-slave-zpbw"
server_key: "4d2e964759ee9075eed277c7d0e195bd"
server_key_iv: "bb9ad9792770e565aad6f0c49d2e9f33"
server_key_version: "encryptionkey@0"
I20260619 15:07:40.217881 10842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:40.232999 10842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:40.237363 10842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:40.237687 10842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:40.238713 10842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:40.240175 10842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:40.240295 10842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:40.240456 10842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:40.240531 10842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:40.275128 10842 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:42999
I20260619 15:07:40.275173 10969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:42999 every 8 connection(s)
I20260619 15:07:40.276520 10842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:07:40.282514 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 10842
I20260619 15:07:40.282739 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:07:40.283380 24989 external_mini_cluster.cc:1428] Setting key 6704bc6d73c4ba5fc4f85dedfacbbf97
Jun 19 15:07:40 dist-test-slave-zpbw krb5kdc[10736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881660, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:07:40.297847 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:40.279219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:54153 (local address 127.24.103.126:46791)
0619 15:07:40.279699 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:40.279704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:40.280610 (+ 906us) server_negotiation.cc:407] Connection header received
0619 15:07:40.282933 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:40.282939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:40.283032 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:40.283152 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:40.285036 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:40.285833 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:40.287557 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:40.287964 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:40.288103 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:40.291951 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:40.291956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:40.291960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:40.291996 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:40.294642 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:40.295449 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:40.295456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:40.295459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:40.295590 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:40.296037 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:40.296042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:40.296045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:40.296319 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:40.296477 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:40.297216 (+ 739us) server_negotiation.cc:299] Negotiation successful
0619 15:07:40.297466 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":253,"threads_started":1}
I20260619 15:07:40.298921 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:40.279679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:46791 (local address 127.24.103.65:54153)
0619 15:07:40.280435 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:40.280481 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:40.282434 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:40.283398 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:40.283407 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:40.284138 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:40.284849 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:40.284869 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:40.286006 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:40.286010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:40.287389 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:40.287402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:40.288211 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:40.288219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:40.288471 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:40.289182 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:07:40.289205 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:40.291709 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:40.294823 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:40.294834 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:40.294865 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:40.295318 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:40.295703 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:40.295708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:40.295712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:40.295913 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:40.296450 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:40.296458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:40.296834 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 15:07:40.297109 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 15:07:40.297684 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":264,"threads_started":1}
I20260619 15:07:40.300648 10970 heartbeater.cc:347] Connected to a master server at 127.24.103.126:46791
I20260619 15:07:40.300993 10970 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:40.301913 10970 heartbeater.cc:511] Master 127.24.103.126:46791 requested a full tablet report, sending...
I20260619 15:07:40.304140 10781 ts_manager.cc:194] Registered new tserver with Master: 8ffffd945938474c810873e349c1112a (127.24.103.65:42999)
I20260619 15:07:40.305577 10781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:54153
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:07:40.338385 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:46791
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:40.681676 10976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:40.681973 10976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:40.682180 10976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:40.690008 10976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:40.690161 10976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:40.690359 10976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:07:40.700392 10976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:46791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:40.701752 10976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:40.703203 10976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:40.716706 10981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:40.716832 10982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:40.716861 10984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:40.717841 10976 server_base.cc:1034] running on GCE node
I20260619 15:07:40.718436 10976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:40.719676 10976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:40.720883 10976 hybrid_clock.cc:648] HybridClock initialized: now 1781881660720825 us; error 53 us; skew 500 ppm
I20260619 15:07:40.721272 10976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:40 dist-test-slave-zpbw krb5kdc[10736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881660, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:40.724656 10976 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:07:40.726181 10976 webserver.cc:460] Webserver started at http://127.24.103.66:36327/ using document root <none> and password file <none>
I20260619 15:07:40.726982 10976 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:40.727135 10976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:40.727464 10976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:40.730695 10976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "868bc41bdb18402bba102b76576d4922"
format_stamp: "Formatted at 2026-06-19 15:07:40 on dist-test-slave-zpbw"
server_key: "a59f4afae16113ef6c91e820d36a5b17"
server_key_iv: "7178cee4630c943d8d2d3f6b17ff0ebe"
server_key_version: "encryptionkey@0"
I20260619 15:07:40.731613 10976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "868bc41bdb18402bba102b76576d4922"
format_stamp: "Formatted at 2026-06-19 15:07:40 on dist-test-slave-zpbw"
server_key: "a59f4afae16113ef6c91e820d36a5b17"
server_key_iv: "7178cee4630c943d8d2d3f6b17ff0ebe"
server_key_version: "encryptionkey@0"
I20260619 15:07:40.737234 10976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:07:40.740968 10991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:40.742354 10976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:07:40.742543 10976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "868bc41bdb18402bba102b76576d4922"
format_stamp: "Formatted at 2026-06-19 15:07:40 on dist-test-slave-zpbw"
server_key: "a59f4afae16113ef6c91e820d36a5b17"
server_key_iv: "7178cee4630c943d8d2d3f6b17ff0ebe"
server_key_version: "encryptionkey@0"
I20260619 15:07:40.742760 10976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:40.757795 10976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:40.763178 10976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:40.763495 10976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:40.764488 10976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:40.765830 10976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:40.765906 10976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:40.765991 10976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:40.766047 10976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:40.802387 10976 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:36419
I20260619 15:07:40.802488 11103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:36419 every 8 connection(s)
I20260619 15:07:40.803805 10976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:07:40.812445 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 10976
I20260619 15:07:40.812656 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:07:40.813117 24989 external_mini_cluster.cc:1428] Setting key 8fb560d0cb4b39c546bbc20af940713d
Jun 19 15:07:40 dist-test-slave-zpbw krb5kdc[10736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881660, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:07:40.825174 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:40.806451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:39139 (local address 127.24.103.126:46791)
0619 15:07:40.806929 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:40.806933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:40.807912 (+ 979us) server_negotiation.cc:407] Connection header received
0619 15:07:40.809780 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:40.809793 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:40.809885 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:40.810002 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:40.811680 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:40.812685 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:40.814341 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:40.814905 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:40.815045 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:40.818948 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:40.818954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:40.818957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:40.818992 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:40.821724 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:40.822589 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:40.822594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:40.822596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:40.822722 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:40.823208 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:40.823212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:40.823214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:40.823515 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:40.823679 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:40.824442 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 15:07:40.824680 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":246,"threads_started":1}
I20260619 15:07:40.826206 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:40.807031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:46791 (local address 127.24.103.66:39139)
0619 15:07:40.807730 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:40.807779 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:40.809489 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:40.810249 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:40.810257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:40.810897 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:40.811499 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:40.811516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:40.812846 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:40.812850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:40.814174 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:40.814186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:40.815150 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:40.815159 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:40.815399 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:40.816130 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:07:40.816152 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:40.818663 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:40.821886 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:40.821896 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:40.821915 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:40.822449 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:40.822867 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:40.822872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:40.822876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:40.823084 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:40.823650 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:40.823659 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:40.824041 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 15:07:40.824325 (+ 284us) client_negotiation.cc:240] Negotiation successful
0619 15:07:40.824986 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":266,"threads_started":1}
I20260619 15:07:40.827791 11104 heartbeater.cc:347] Connected to a master server at 127.24.103.126:46791
I20260619 15:07:40.828111 11104 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:40.828855 11104 heartbeater.cc:511] Master 127.24.103.126:46791 requested a full tablet report, sending...
I20260619 15:07:40.830435 10781 ts_manager.cc:194] Registered new tserver with Master: 868bc41bdb18402bba102b76576d4922 (127.24.103.66:36419)
I20260619 15:07:40.831772 10781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:39139
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:07:40.868296 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:46791
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:41.222551 11110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:41.222913 11110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:41.223129 11110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:41.230505 11110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:41.230683 11110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:41.230907 11110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:07:41.241086 11110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:46791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:41.242556 11110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:41.243973 11110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:41.258073 11116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:41.258100 11115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:41.258031 11118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:41.258878 11110 server_base.cc:1034] running on GCE node
I20260619 15:07:41.259586 11110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:41.260593 11110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:41.261827 11110 hybrid_clock.cc:648] HybridClock initialized: now 1781881661261779 us; error 66 us; skew 500 ppm
I20260619 15:07:41.262149 11110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:41 dist-test-slave-zpbw krb5kdc[10736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881661, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:41.265381 11110 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:07:41.266944 11110 webserver.cc:460] Webserver started at http://127.24.103.67:40595/ using document root <none> and password file <none>
I20260619 15:07:41.267661 11110 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:41.267817 11110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:41.268139 11110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:41.271174 11110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "1162708b533e40b3aca74b5072782522"
format_stamp: "Formatted at 2026-06-19 15:07:41 on dist-test-slave-zpbw"
server_key: "6bffdce28abc924fe4b42db232abe53d"
server_key_iv: "415cca7105b11ca316188867ceb0ed44"
server_key_version: "encryptionkey@0"
I20260619 15:07:41.271965 11110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "1162708b533e40b3aca74b5072782522"
format_stamp: "Formatted at 2026-06-19 15:07:41 on dist-test-slave-zpbw"
server_key: "6bffdce28abc924fe4b42db232abe53d"
server_key_iv: "415cca7105b11ca316188867ceb0ed44"
server_key_version: "encryptionkey@0"
I20260619 15:07:41.277608 11110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:07:41.281324 11125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:41.282853 11110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 15:07:41.283034 11110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "1162708b533e40b3aca74b5072782522"
format_stamp: "Formatted at 2026-06-19 15:07:41 on dist-test-slave-zpbw"
server_key: "6bffdce28abc924fe4b42db232abe53d"
server_key_iv: "415cca7105b11ca316188867ceb0ed44"
server_key_version: "encryptionkey@0"
I20260619 15:07:41.283243 11110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:41.307109 11110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:41.308869 10970 heartbeater.cc:503] Master 127.24.103.126:46791 was elected leader, sending a full tablet report...
I20260619 15:07:41.312260 11110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:41.312525 11110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:41.313478 11110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:41.314883 11110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:41.314963 11110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:41.315064 11110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:41.315109 11110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:41.347532 11110 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:34035
I20260619 15:07:41.347584 11237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:34035 every 8 connection(s)
I20260619 15:07:41.349151 11110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:07:41.357443 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 11110
I20260619 15:07:41.357645 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:07:41.358183 24989 external_mini_cluster.cc:1428] Setting key 41d5f6c8a096b865ce9e07981881cf17
Jun 19 15:07:41 dist-test-slave-zpbw krb5kdc[10736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881661, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:07:41.369776 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.351799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:49881 (local address 127.24.103.126:46791)
0619 15:07:41.352241 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:41.352246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:41.353207 (+ 961us) server_negotiation.cc:407] Connection header received
0619 15:07:41.355136 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:41.355142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:41.355251 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:41.355372 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:41.356879 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.358097 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.360244 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.360671 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.360773 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.364362 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:41.364388 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:41.364395 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:41.364443 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:41.367172 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:41.367840 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:41.367844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:41.367847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:41.367958 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:41.368340 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:41.368343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:41.368345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:41.368597 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:41.368711 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:41.369351 (+ 640us) server_negotiation.cc:299] Negotiation successful
0619 15:07:41.369510 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":211,"threads_started":1}
I20260619 15:07:41.370766 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.352364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:46791 (local address 127.24.103.67:49881)
0619 15:07:41.353046 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:41.353083 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:41.354860 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:41.355619 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:41.355628 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:41.356134 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:41.356705 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:41.356722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.358300 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.358305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:41.360098 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:41.360112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.360911 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.360918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:41.361228 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.361912 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:07:41.361935 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:41.364153 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:41.367303 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:41.367309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:41.367322 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:41.367732 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:41.368056 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:41.368059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:41.368061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:41.368235 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:41.368705 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:41.368711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:41.369020 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 15:07:41.369242 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 15:07:41.369642 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":276,"threads_started":1}
I20260619 15:07:41.372382 11238 heartbeater.cc:347] Connected to a master server at 127.24.103.126:46791
I20260619 15:07:41.372713 11238 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:41.373639 11238 heartbeater.cc:511] Master 127.24.103.126:46791 requested a full tablet report, sending...
I20260619 15:07:41.375051 10781 ts_manager.cc:194] Registered new tserver with Master: 1162708b533e40b3aca74b5072782522 (127.24.103.67:34035)
I20260619 15:07:41.375818 10781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:49881
I20260619 15:07:41.386264 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:07:41.403388 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.390449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.24.103.126:46791)
0619 15:07:41.390612 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:41.390616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:41.391094 (+ 478us) server_negotiation.cc:407] Connection header received
0619 15:07:41.391239 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:41.391243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:41.391325 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:41.391407 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:41.392819 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.393555 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.394982 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.395401 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.395499 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.398105 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:41.398110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:41.398114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:41.398148 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:41.400674 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:41.401508 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:41.401512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:41.401514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:41.401635 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:41.402006 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:41.402009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:41.402011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:41.402321 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:41.402519 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:41.402931 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 15:07:41.403092 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 15:07:41.407733 10781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:07:41.414705 10781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:07:41.451741 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.427426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:34035 (local address 127.0.0.1:56378)
0619 15:07:41.440060 (+ 12634us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:41.440108 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:41.440356 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:41.441314 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:41.441325 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:41.441377 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:41.442037 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:41.442056 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.445967 (+ 3911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.445975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:41.448711 (+ 2736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:41.448725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.450508 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.450515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:41.450760 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.450879 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 15:07:41.451026 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 15:07:41.451243 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12352,"mutex_wait_us":112,"thread_start_us":12167,"threads_started":1}
I20260619 15:07:41.451910 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.427593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.24.103.67:34035)
0619 15:07:41.439614 (+ 12021us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:41.439628 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:41.440577 (+ 949us) server_negotiation.cc:407] Connection header received
0619 15:07:41.440710 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:41.440718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:41.441022 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:41.442295 (+ 1273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:41.442854 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.445795 (+ 2941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.448923 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.450267 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.450417 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.450673 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:41.451310 (+ 637us) server_negotiation.cc:299] Negotiation successful
0619 15:07:41.451490 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11750,"thread_start_us":11512,"threads_started":1}
I20260619 15:07:41.452793 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.428034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.24.103.66:36419)
0619 15:07:41.440734 (+ 12700us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:41.440748 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:41.441769 (+ 1021us) server_negotiation.cc:407] Connection header received
0619 15:07:41.441878 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:41.441885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:41.442122 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:41.442302 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:41.443313 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.445247 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.449275 (+ 4028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.450873 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.451011 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.451197 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:41.452272 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0619 15:07:41.452466 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12469,"thread_start_us":12732,"threads_started":1}
I20260619 15:07:41.453469 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.427558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:36419 (local address 127.0.0.1:44150)
0619 15:07:41.441466 (+ 13908us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:41.441486 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:41.441625 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:41.442524 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:41.442528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:41.442550 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:41.443036 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:41.443054 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.446412 (+ 3358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.446418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:41.448864 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:41.448884 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.451094 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.451100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:41.451313 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.451346 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 15:07:41.451434 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 15:07:41.453095 (+ 1661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13780,"mutex_wait_us":25,"thread_start_us":13629,"threads_started":1}
I20260619 15:07:41.455163 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.442649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42999 (local address 127.0.0.1:47738)
0619 15:07:41.446533 (+ 3884us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:41.446552 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:41.446677 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:41.447383 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:41.447388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:41.447416 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:41.447940 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:41.447953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.449812 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.449817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:41.452502 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:41.452515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.454318 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.454328 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:41.454502 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.454532 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 15:07:41.454636 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 15:07:41.454756 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3735,"mutex_wait_us":9,"thread_start_us":3497,"threads_started":1}
I20260619 15:07:41.455865 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.441108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.24.103.65:42999)
0619 15:07:41.446352 (+ 5244us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:41.446363 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:41.446638 (+ 275us) server_negotiation.cc:407] Connection header received
0619 15:07:41.446868 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:41.446878 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:41.447174 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:41.447379 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:41.448112 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.449646 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.452659 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.454079 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.454251 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.454499 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:41.455375 (+ 876us) server_negotiation.cc:299] Negotiation successful
0619 15:07:41.455546 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4754,"thread_start_us":4618,"threads_started":1}
I20260619 15:07:41.456440 11173 tablet_service.cc:1467] Processing CreateTablet for tablet 485c399259dd412e9ebfe4cf0f5c4ddf (DEFAULT_TABLE table=test-table [id=34c5e54dd5674f3db9b7869c321ee761]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:41.457722 11173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485c399259dd412e9ebfe4cf0f5c4ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:41.457441 11039 tablet_service.cc:1467] Processing CreateTablet for tablet 485c399259dd412e9ebfe4cf0f5c4ddf (DEFAULT_TABLE table=test-table [id=34c5e54dd5674f3db9b7869c321ee761]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:41.458662 11039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485c399259dd412e9ebfe4cf0f5c4ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:41.462707 10905 tablet_service.cc:1467] Processing CreateTablet for tablet 485c399259dd412e9ebfe4cf0f5c4ddf (DEFAULT_TABLE table=test-table [id=34c5e54dd5674f3db9b7869c321ee761]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:41.464430 10905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485c399259dd412e9ebfe4cf0f5c4ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:41.469910 11257 tablet_bootstrap.cc:492] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522: Bootstrap starting.
I20260619 15:07:41.473309 11257 tablet_bootstrap.cc:654] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:41.474526 11257 log.cc:824] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:41.475672 11258 tablet_bootstrap.cc:492] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922: Bootstrap starting.
I20260619 15:07:41.479358 11257 tablet_bootstrap.cc:492] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522: No bootstrap required, opened a new log
I20260619 15:07:41.479559 11258 tablet_bootstrap.cc:654] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:41.479669 11257 ts_tablet_manager.cc:1397] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260619 15:07:41.480955 11258 log.cc:824] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:41.481607 11260 tablet_bootstrap.cc:492] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a: Bootstrap starting.
I20260619 15:07:41.485641 11260 tablet_bootstrap.cc:654] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:41.486749 11260 log.cc:824] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:41.489990 11258 tablet_bootstrap.cc:492] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922: No bootstrap required, opened a new log
I20260619 15:07:41.490316 11258 ts_tablet_manager.cc:1397] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260619 15:07:41.491777 11257 raft_consensus.cc:348] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1162708b533e40b3aca74b5072782522" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34035 } } peers { permanent_uuid: "868bc41bdb18402bba102b76576d4922" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36419 } } peers { permanent_uuid: "8ffffd945938474c810873e349c1112a" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42999 } }
I20260619 15:07:41.492251 11257 raft_consensus.cc:374] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:41.492379 11257 raft_consensus.cc:729] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1162708b533e40b3aca74b5072782522, State: Initialized, Role: FOLLOWER
I20260619 15:07:41.493106 11260 tablet_bootstrap.cc:492] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a: No bootstrap required, opened a new log
I20260619 15:07:41.493034 11257 consensus_queue.cc:260] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1162708b533e40b3aca74b5072782522" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34035 } } peers { permanent_uuid: "868bc41bdb18402bba102b76576d4922" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36419 } } peers { permanent_uuid: "8ffffd945938474c810873e349c1112a" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42999 } }
I20260619 15:07:41.493487 11260 ts_tablet_manager.cc:1397] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260619 15:07:41.499666 11238 heartbeater.cc:503] Master 127.24.103.126:46791 was elected leader, sending a full tablet report...
I20260619 15:07:41.500463 11257 ts_tablet_manager.cc:1428] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522: Time spent starting tablet: real 0.021s user 0.010s sys 0.010s
I20260619 15:07:41.501907 11260 raft_consensus.cc:348] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1162708b533e40b3aca74b5072782522" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34035 } } peers { permanent_uuid: "868bc41bdb18402bba102b76576d4922" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36419 } } peers { permanent_uuid: "8ffffd945938474c810873e349c1112a" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42999 } }
I20260619 15:07:41.502317 11260 raft_consensus.cc:374] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:41.502441 11260 raft_consensus.cc:729] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ffffd945938474c810873e349c1112a, State: Initialized, Role: FOLLOWER
I20260619 15:07:41.503157 11260 consensus_queue.cc:260] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1162708b533e40b3aca74b5072782522" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34035 } } peers { permanent_uuid: "868bc41bdb18402bba102b76576d4922" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36419 } } peers { permanent_uuid: "8ffffd945938474c810873e349c1112a" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42999 } }
I20260619 15:07:41.503116 11258 raft_consensus.cc:348] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1162708b533e40b3aca74b5072782522" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34035 } } peers { permanent_uuid: "868bc41bdb18402bba102b76576d4922" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36419 } } peers { permanent_uuid: "8ffffd945938474c810873e349c1112a" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42999 } }
I20260619 15:07:41.503618 11258 raft_consensus.cc:374] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:41.503726 11258 raft_consensus.cc:729] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868bc41bdb18402bba102b76576d4922, State: Initialized, Role: FOLLOWER
I20260619 15:07:41.504706 11258 consensus_queue.cc:260] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1162708b533e40b3aca74b5072782522" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34035 } } peers { permanent_uuid: "868bc41bdb18402bba102b76576d4922" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36419 } } peers { permanent_uuid: "8ffffd945938474c810873e349c1112a" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42999 } }
I20260619 15:07:41.505864 11260 ts_tablet_manager.cc:1428] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 15:07:41.506991 11104 heartbeater.cc:503] Master 127.24.103.126:46791 was elected leader, sending a full tablet report...
I20260619 15:07:41.508847 11258 ts_tablet_manager.cc:1428] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922: Time spent starting tablet: real 0.018s user 0.010s sys 0.007s
W20260619 15:07:41.529021 10971 tablet.cc:2369] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:07:41.555860 11105 tablet.cc:2369] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:07:41.595022 11264 raft_consensus.cc:482] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:07:41.595389 11264 raft_consensus.cc:504] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1162708b533e40b3aca74b5072782522" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34035 } } peers { permanent_uuid: "868bc41bdb18402bba102b76576d4922" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36419 } } peers { permanent_uuid: "8ffffd945938474c810873e349c1112a" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42999 } }
I20260619 15:07:41.597230 11264 leader_election.cc:288] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1162708b533e40b3aca74b5072782522 (127.24.103.67:34035), 868bc41bdb18402bba102b76576d4922 (127.24.103.66:36419)
W20260619 15:07:41.600816 11239 tablet.cc:2369] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:07:41.606194 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.597658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:49219 (local address 127.24.103.67:34035)
0619 15:07:41.597882 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:41.597890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:41.598959 (+ 1069us) server_negotiation.cc:407] Connection header received
0619 15:07:41.599202 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:41.599207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:41.599293 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:41.599385 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:41.600174 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.601044 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.603796 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.605051 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.605228 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.605307 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:41.605793 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 15:07:41.605923 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 15:07:41.606192 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.597944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:51307 (local address 127.24.103.66:36419)
0619 15:07:41.598213 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:41.598221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:41.598915 (+ 694us) server_negotiation.cc:407] Connection header received
0619 15:07:41.599286 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:41.599298 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:41.599386 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:41.599473 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:41.600309 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.601348 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.603789 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.605049 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.605228 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.605314 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:41.605786 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 15:07:41.605923 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 15:07:41.606421 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.597664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:34035 (local address 127.24.103.65:49219)
0619 15:07:41.598768 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:41.598849 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:41.599027 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:41.599419 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:41.599427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:41.599458 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:41.599945 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:41.599957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.601200 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.601204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:41.603557 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:41.603578 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.605359 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.605364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:41.605572 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.605642 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 15:07:41.605745 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 15:07:41.605902 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"mutex_wait_us":31,"thread_start_us":407,"threads_started":1}
I20260619 15:07:41.606447 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.597519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:36419 (local address 127.24.103.65:51307)
0619 15:07:41.598807 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:41.598829 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:41.599110 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:41.599573 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:41.599577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:41.599615 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:41.600152 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:41.600166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.601495 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.601499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:41.603638 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:41.603653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.605347 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.605354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:41.605575 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.605642 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 15:07:41.605745 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 15:07:41.605902 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":62,"spinlock_wait_cycles":34560,"thread_start_us":948,"threads_started":1}
I20260619 15:07:41.607583 11193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485c399259dd412e9ebfe4cf0f5c4ddf" candidate_uuid: "8ffffd945938474c810873e349c1112a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1162708b533e40b3aca74b5072782522" is_pre_election: true
I20260619 15:07:41.607577 11059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485c399259dd412e9ebfe4cf0f5c4ddf" candidate_uuid: "8ffffd945938474c810873e349c1112a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868bc41bdb18402bba102b76576d4922" is_pre_election: true
I20260619 15:07:41.608069 11059 raft_consensus.cc:2454] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ffffd945938474c810873e349c1112a in term 0.
I20260619 15:07:41.608069 11193 raft_consensus.cc:2454] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ffffd945938474c810873e349c1112a in term 0.
I20260619 15:07:41.608799 10859 leader_election.cc:302] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1162708b533e40b3aca74b5072782522, 868bc41bdb18402bba102b76576d4922, 8ffffd945938474c810873e349c1112a; no voters:
I20260619 15:07:41.609230 11264 raft_consensus.cc:2781] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:07:41.609390 11264 raft_consensus.cc:482] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:07:41.609516 11264 raft_consensus.cc:3037] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:41.611793 11264 raft_consensus.cc:504] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1162708b533e40b3aca74b5072782522" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34035 } } peers { permanent_uuid: "868bc41bdb18402bba102b76576d4922" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36419 } } peers { permanent_uuid: "8ffffd945938474c810873e349c1112a" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42999 } }
I20260619 15:07:41.612516 11264 leader_election.cc:288] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [CANDIDATE]: Term 1 election: Requested vote from peers 1162708b533e40b3aca74b5072782522 (127.24.103.67:34035), 868bc41bdb18402bba102b76576d4922 (127.24.103.66:36419)
I20260619 15:07:41.612967 11193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485c399259dd412e9ebfe4cf0f5c4ddf" candidate_uuid: "8ffffd945938474c810873e349c1112a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1162708b533e40b3aca74b5072782522"
I20260619 15:07:41.613085 11059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485c399259dd412e9ebfe4cf0f5c4ddf" candidate_uuid: "8ffffd945938474c810873e349c1112a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868bc41bdb18402bba102b76576d4922"
I20260619 15:07:41.613183 11193 raft_consensus.cc:3037] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:41.613340 11059 raft_consensus.cc:3037] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:41.615517 11193 raft_consensus.cc:2454] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ffffd945938474c810873e349c1112a in term 1.
I20260619 15:07:41.615674 11059 raft_consensus.cc:2454] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ffffd945938474c810873e349c1112a in term 1.
I20260619 15:07:41.615996 10860 leader_election.cc:302] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1162708b533e40b3aca74b5072782522, 8ffffd945938474c810873e349c1112a; no voters:
I20260619 15:07:41.616324 11264 raft_consensus.cc:2781] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:41.616794 11264 raft_consensus.cc:686] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 1 LEADER]: Becoming Leader. State: Replica: 8ffffd945938474c810873e349c1112a, State: Running, Role: LEADER
I20260619 15:07:41.617352 11264 consensus_queue.cc:237] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1162708b533e40b3aca74b5072782522" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34035 } } peers { permanent_uuid: "868bc41bdb18402bba102b76576d4922" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36419 } } peers { permanent_uuid: "8ffffd945938474c810873e349c1112a" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42999 } }
I20260619 15:07:41.621833 10779 catalog_manager.cc:5495] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ffffd945938474c810873e349c1112a (127.24.103.65). New cstate: current_term: 1 leader_uuid: "8ffffd945938474c810873e349c1112a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1162708b533e40b3aca74b5072782522" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "868bc41bdb18402bba102b76576d4922" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ffffd945938474c810873e349c1112a" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42999 } health_report { overall_health: HEALTHY } } }
I20260619 15:07:41.650420 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.644849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.24.103.65:42999)
0619 15:07:41.645056 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:41.645062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:41.645106 (+ 44us) server_negotiation.cc:407] Connection header received
0619 15:07:41.645339 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:41.645344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:41.645446 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:41.645636 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:07:41.645965 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.646768 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.648352 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.648770 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.648881 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.649382 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:07:41.649893 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:07:41.650033 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:41.650150 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 15:07:41.650204 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 15:07:41.659413 11059 raft_consensus.cc:1261] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922 [term 1 FOLLOWER]: Refusing update from remote peer 8ffffd945938474c810873e349c1112a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:07:41.659416 11193 raft_consensus.cc:1261] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522 [term 1 FOLLOWER]: Refusing update from remote peer 8ffffd945938474c810873e349c1112a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:07:41.660455 11268 consensus_queue.cc:1035] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1162708b533e40b3aca74b5072782522" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:07:41.660790 11264 consensus_queue.cc:1035] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [LEADER]: Connected to new peer: Peer: permanent_uuid: "868bc41bdb18402bba102b76576d4922" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:07:41.689249 10779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:07:41.689546 10779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:07:41.692994 10779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9885e61d825b4db49b446d48dc1d3f8a: Sending DeleteTablet for 3 replicas of tablet 485c399259dd412e9ebfe4cf0f5c4ddf
I20260619 15:07:41.694263 11039 tablet_service.cc:1514] Processing DeleteTablet for tablet 485c399259dd412e9ebfe4cf0f5c4ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:41 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:44150
I20260619 15:07:41.694350 10905 tablet_service.cc:1514] Processing DeleteTablet for tablet 485c399259dd412e9ebfe4cf0f5c4ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:41 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:47738
I20260619 15:07:41.694612 11173 tablet_service.cc:1514] Processing DeleteTablet for tablet 485c399259dd412e9ebfe4cf0f5c4ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:41 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:56378
I20260619 15:07:41.697332 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 10842
I20260619 15:07:41.705827 11282 tablet_replica.cc:330] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922: stopping tablet replica
I20260619 15:07:41.706463 11284 tablet_replica.cc:330] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522: stopping tablet replica
I20260619 15:07:41.707222 11282 raft_consensus.cc:2229] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:07:41.707705 11284 raft_consensus.cc:2229] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:07:41.708316 11284 raft_consensus.cc:2258] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:41.708827 11282 raft_consensus.cc:2258] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:41.711835 11284 ts_tablet_manager.cc:1905] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:41.714087 11282 ts_tablet_manager.cc:1905] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 15:07:41 dist-test-slave-zpbw krb5kdc[10736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:07:41.721683 11282 ts_tablet_manager.cc:1918] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:41.729372 11284 ts_tablet_manager.cc:1918] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:41.730151 11283 tablet_replica.cc:330] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a: stopping tablet replica
I20260619 15:07:41.730669 11283 raft_consensus.cc:2229] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:07:41.731420 11283 raft_consensus.cc:2258] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:41.732389 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.699907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.24.103.65:42999)
0619 15:07:41.700995 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:41.701005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:41.703572 (+ 2567us) server_negotiation.cc:407] Connection header received
0619 15:07:41.703692 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:41.703701 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:41.703826 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:41.703955 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:41.708588 (+ 4633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.710293 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.713011 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.714086 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.714224 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.722329 (+ 8105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:41.722358 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:41.722386 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:41.722473 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:41.728307 (+ 5834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:41.729720 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:41.729739 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:41.729746 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:41.729964 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:41.730464 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:41.730469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:41.730472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:41.730996 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:41.731412 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:41.731671 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 15:07:41.731950 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796}
I20260619 15:07:41.733222 10965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:41.734141 11284 log.cc:1196] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/485c399259dd412e9ebfe4cf0f5c4ddf
I20260619 15:07:41.734143 11282 log.cc:1196] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/485c399259dd412e9ebfe4cf0f5c4ddf
I20260619 15:07:41.734956 11284 ts_tablet_manager.cc:1939] T 485c399259dd412e9ebfe4cf0f5c4ddf P 1162708b533e40b3aca74b5072782522: Deleting consensus metadata
I20260619 15:07:41.734998 11282 ts_tablet_manager.cc:1939] T 485c399259dd412e9ebfe4cf0f5c4ddf P 868bc41bdb18402bba102b76576d4922: Deleting consensus metadata
I20260619 15:07:41.737003 10766 catalog_manager.cc:4841] TS 868bc41bdb18402bba102b76576d4922 (127.24.103.66:36419): tablet 485c399259dd412e9ebfe4cf0f5c4ddf (table test-table [id=34c5e54dd5674f3db9b7869c321ee761]) successfully deleted
I20260619 15:07:41.737053 10767 catalog_manager.cc:4841] TS 1162708b533e40b3aca74b5072782522 (127.24.103.67:34035): tablet 485c399259dd412e9ebfe4cf0f5c4ddf (table test-table [id=34c5e54dd5674f3db9b7869c321ee761]) successfully deleted
I20260619 15:07:41.733932 11283 ts_tablet_manager.cc:1905] T 485c399259dd412e9ebfe4cf0f5c4ddf P 8ffffd945938474c810873e349c1112a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:41.839480 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 10842
W20260619 15:07:41.858757 10767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.103.65:42999 (error 108)
I20260619 15:07:41.859061 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 10976
I20260619 15:07:41.860882 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.860396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42999 (local address 127.0.0.1:47768)
0619 15:07:41.860583 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:41.860706 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:42999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 15:07:41.861166 10767 catalog_manager.cc:4584] TS 8ffffd945938474c810873e349c1112a (127.24.103.65:42999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 485c399259dd412e9ebfe4cf0f5c4ddf: Network error: Client connection negotiation failed: client connection to 127.24.103.65:42999: connect: Connection refused (error 111)
Jun 19 15:07:41 dist-test-slave-zpbw krb5kdc[10736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:07:41.875219 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.859987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.24.103.66:36419)
0619 15:07:41.860160 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:41.860164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:41.860183 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:07:41.860378 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:41.860384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:41.860501 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:41.860605 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:41.861978 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.862742 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.864004 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.864387 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.864476 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.869305 (+ 4829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:41.869323 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:41.869339 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:41.869376 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:41.872457 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:41.873244 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:41.873252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:41.873255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:41.873385 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:41.873731 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:41.873735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:41.873737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:41.874120 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:41.874355 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:41.874590 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 15:07:41.874765 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:07:41.875979 11099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:41.915141 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.914479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42999 (local address 127.0.0.1:47778)
0619 15:07:41.914753 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:41.914938 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:42999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 15:07:41.916227 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.915838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42999 (local address 127.0.0.1:47784)
0619 15:07:41.915996 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:41.916092 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:42999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 15:07:41.968070 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 10976
I20260619 15:07:41.976814 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.976326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42999 (local address 127.0.0.1:47788)
0619 15:07:41.976558 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:41.976649 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:42999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 15:07:41.977957 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.977627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42999 (local address 127.0.0.1:47800)
0619 15:07:41.977761 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:41.977842 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:42999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 15:07:41.987051 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 11110
Jun 19 15:07:41 dist-test-slave-zpbw krb5kdc[10736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:07:42.002529 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:41.987794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.24.103.67:34035)
0619 15:07:41.987991 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:41.987996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:41.988018 (+ 22us) server_negotiation.cc:407] Connection header received
0619 15:07:41.988228 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:41.988232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:41.988318 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:41.988401 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:41.989689 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.990428 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.991810 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:41.992214 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:41.992315 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:41.996950 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:41.996970 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:41.996986 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:41.997024 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:41.999743 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:42.000508 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:42.000516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:42.000518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:42.000661 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:42.001017 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:42.001024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:42.001029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:42.001476 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:42.001700 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:42.001975 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 15:07:42.002148 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:07:42.003219 11233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:42.058749 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:42.058224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42999 (local address 127.0.0.1:47808)
0619 15:07:42.058456 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:42.058576 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:42999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 15:07:42.059904 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:42.059521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42999 (local address 127.0.0.1:47824)
0619 15:07:42.059716 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:42.059798 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:42999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 15:07:42.097795 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 11110
I20260619 15:07:42.114480 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 10749
I20260619 15:07:42.115680 10811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:42.227279 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 10749
2026-06-19T15:07:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4416 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T15:07:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:07:42Z Disabled control of system clock
I20260619 15:07:42.305718 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35263
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:35263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:07:42.639462 11294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:42.639786 11294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:42.639869 11294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:42.647397 11294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:07:42.647536 11294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:42.647575 11294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:42.647631 11294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:07:42.647677 11294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:07:42.657814 11294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:35263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:42.659224 11294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:42.661000 11294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:42.673466 11300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:42.673542 11299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:42.673578 11302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:42.674512 11294 server_base.cc:1034] running on GCE node
I20260619 15:07:42.675450 11294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:42.676955 11294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:42.678200 11294 hybrid_clock.cc:648] HybridClock initialized: now 1781881662678158 us; error 31 us; skew 500 ppm
I20260619 15:07:42.678532 11294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:07:42.680972 11294 webserver.cc:460] Webserver started at http://127.24.103.126:40507/ using document root <none> and password file <none>
I20260619 15:07:42.681664 11294 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:42.681751 11294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:42.682027 11294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:42.685133 11294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "2e808529a7c341d4a4142d2b5f4dfa67"
format_stamp: "Formatted at 2026-06-19 15:07:42 on dist-test-slave-zpbw"
server_key: "974997fd3246a3251d004f79f0e8c999"
server_key_iv: "02bca87d7587b28472daf43f7fd62886"
server_key_version: "encryptionkey@0"
I20260619 15:07:42.685928 11294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "2e808529a7c341d4a4142d2b5f4dfa67"
format_stamp: "Formatted at 2026-06-19 15:07:42 on dist-test-slave-zpbw"
server_key: "974997fd3246a3251d004f79f0e8c999"
server_key_iv: "02bca87d7587b28472daf43f7fd62886"
server_key_version: "encryptionkey@0"
I20260619 15:07:42.691530 11294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 15:07:42.695257 11308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:42.696759 11294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:07:42.696923 11294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "2e808529a7c341d4a4142d2b5f4dfa67"
format_stamp: "Formatted at 2026-06-19 15:07:42 on dist-test-slave-zpbw"
server_key: "974997fd3246a3251d004f79f0e8c999"
server_key_iv: "02bca87d7587b28472daf43f7fd62886"
server_key_version: "encryptionkey@0"
I20260619 15:07:42.697116 11294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:42.719249 11294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:42.720130 11294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:42.720396 11294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:42.739521 11294 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:35263
I20260619 15:07:42.739540 11359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:35263 every 8 connection(s)
I20260619 15:07:42.741075 11294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:07:42.745038 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:42.751025 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 11294
I20260619 15:07:42.751214 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:07:42.751667 24989 external_mini_cluster.cc:1428] Setting key bd63bdd7186c890f372a6553dac2e3b3
I20260619 15:07:42.756579 11360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67: Bootstrap starting.
I20260619 15:07:42.760392 11360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:42.761765 11360 log.cc:824] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:42.765375 11360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67: No bootstrap required, opened a new log
I20260619 15:07:42.767163 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:42.753800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58654 (local address 127.24.103.126:35263)
0619 15:07:42.755014 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:42.755026 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:42.755072 (+ 46us) server_negotiation.cc:407] Connection header received
0619 15:07:42.756899 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:42.756923 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:42.757426 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:42.757748 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:42.758489 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:42.759892 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:42.762005 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:42.763199 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:42.763389 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:42.764647 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:42.764655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:42.764670 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:42.765227 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:42.765315 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:42.765336 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:42.765421 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:42.765730 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 15:07:42.766103 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":433,"threads_started":1}
I20260619 15:07:42.773824 11360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e808529a7c341d4a4142d2b5f4dfa67" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35263 } }
I20260619 15:07:42.774356 11360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:42.774482 11360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e808529a7c341d4a4142d2b5f4dfa67, State: Initialized, Role: FOLLOWER
I20260619 15:07:42.775219 11360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e808529a7c341d4a4142d2b5f4dfa67" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35263 } }
I20260619 15:07:42.775449 11360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:07:42.775553 11360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:07:42.775679 11360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:42.778416 11360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e808529a7c341d4a4142d2b5f4dfa67" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35263 } }
I20260619 15:07:42.779126 11360 leader_election.cc:302] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e808529a7c341d4a4142d2b5f4dfa67; no voters:
I20260619 15:07:42.780439 11360 leader_election.cc:288] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:07:42.780632 11365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:42.781358 11365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [term 1 LEADER]: Becoming Leader. State: Replica: 2e808529a7c341d4a4142d2b5f4dfa67, State: Running, Role: LEADER
I20260619 15:07:42.782011 11365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e808529a7c341d4a4142d2b5f4dfa67" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35263 } }
I20260619 15:07:42.782981 11360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:07:42.785085 11366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e808529a7c341d4a4142d2b5f4dfa67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e808529a7c341d4a4142d2b5f4dfa67" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35263 } } }
I20260619 15:07:42.785562 11366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:42.785106 11367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e808529a7c341d4a4142d2b5f4dfa67. Latest consensus state: current_term: 1 leader_uuid: "2e808529a7c341d4a4142d2b5f4dfa67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e808529a7c341d4a4142d2b5f4dfa67" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35263 } } }
I20260619 15:07:42.786197 11367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:42.787307 11370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:07:42.792740 11370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:07:42.802755 11370 catalog_manager.cc:1347] Generated new cluster ID: 20f56b4a6bbd482d93efd683362e5d3d
I20260619 15:07:42.802919 11370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:07:42.830349 11370 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:07:42.831825 11370 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:07:42.847186 11370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67: Generated new TSK 0
I20260619 15:07:42.848104 11370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:07:42.858325 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35263
--builtin_ntp_servers=127.24.103.84:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:07:43.213101 11384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:43.213398 11384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:43.213483 11384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:43.221067 11384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:43.221189 11384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:43.221362 11384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:07:43.230741 11384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:43.232156 11384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:43.234045 11384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:43.248075 11390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:43.248157 11392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:43.248138 11389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:43.248952 11384 server_base.cc:1034] running on GCE node
I20260619 15:07:43.249711 11384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:43.250730 11384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:43.252002 11384 hybrid_clock.cc:648] HybridClock initialized: now 1781881663251953 us; error 44 us; skew 500 ppm
I20260619 15:07:43.252362 11384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:07:43.254767 11384 webserver.cc:460] Webserver started at http://127.24.103.65:45167/ using document root <none> and password file <none>
I20260619 15:07:43.255604 11384 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:43.255740 11384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:43.256091 11384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:43.259112 11384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "260781d0e0cd4f12bc6e16528554e4dd"
format_stamp: "Formatted at 2026-06-19 15:07:43 on dist-test-slave-zpbw"
server_key: "861c06fd6430ccd9d262f44154458b35"
server_key_iv: "1d6181baae856753e9102a6ea165f48d"
server_key_version: "encryptionkey@0"
I20260619 15:07:43.259954 11384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "260781d0e0cd4f12bc6e16528554e4dd"
format_stamp: "Formatted at 2026-06-19 15:07:43 on dist-test-slave-zpbw"
server_key: "861c06fd6430ccd9d262f44154458b35"
server_key_iv: "1d6181baae856753e9102a6ea165f48d"
server_key_version: "encryptionkey@0"
I20260619 15:07:43.265503 11384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:07:43.269359 11398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:43.270979 11384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 15:07:43.271169 11384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "260781d0e0cd4f12bc6e16528554e4dd"
format_stamp: "Formatted at 2026-06-19 15:07:43 on dist-test-slave-zpbw"
server_key: "861c06fd6430ccd9d262f44154458b35"
server_key_iv: "1d6181baae856753e9102a6ea165f48d"
server_key_version: "encryptionkey@0"
I20260619 15:07:43.271402 11384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:43.306516 11384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:43.307484 11384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:43.307808 11384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:43.308887 11384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:43.310413 11384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:43.310542 11384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:43.310668 11384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:43.310756 11384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:43.345109 11384 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:44803
I20260619 15:07:43.345206 11510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:44803 every 8 connection(s)
I20260619 15:07:43.346661 11384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:07:43.348301 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 11384
I20260619 15:07:43.348502 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:07:43.349009 24989 external_mini_cluster.cc:1428] Setting key ac362cd74e1ae6f3f848de6b7e6fa11f
I20260619 15:07:43.352207 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35263
--builtin_ntp_servers=127.24.103.84:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 15:07:43.363149 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:43.349763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:39593 (local address 127.24.103.126:35263)
0619 15:07:43.350243 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:43.350250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:43.351822 (+ 1572us) server_negotiation.cc:407] Connection header received
0619 15:07:43.354867 (+ 3045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:43.354872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:43.354943 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:43.355039 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:43.356499 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:43.357206 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:43.359847 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:43.360244 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:43.360354 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:43.362082 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:43.362091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:43.362099 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:43.362128 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:43.362182 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:43.362198 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:43.362310 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:43.362514 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 15:07:43.362637 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":241,"threads_started":1}
I20260619 15:07:43.364046 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:43.350500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35263 (local address 127.24.103.65:39593)
0619 15:07:43.351642 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:43.351696 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:43.354468 (+ 2772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:43.355328 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:43.355343 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:43.355475 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:43.356318 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:43.356346 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:43.357391 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:43.357400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:43.359686 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:43.359707 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:43.360498 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:43.360508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:43.360865 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:43.361716 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:07:43.361719 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:43.361762 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:07:43.361854 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:43.362308 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:43.362316 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:43.362322 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 15:07:43.362469 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 15:07:43.362854 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":353,"threads_started":1}
I20260619 15:07:43.365731 11511 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35263
I20260619 15:07:43.366078 11511 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:43.367049 11511 heartbeater.cc:511] Master 127.24.103.126:35263 requested a full tablet report, sending...
I20260619 15:07:43.369338 11325 ts_manager.cc:194] Registered new tserver with Master: 260781d0e0cd4f12bc6e16528554e4dd (127.24.103.65:44803)
I20260619 15:07:43.371013 11325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.65:39593
W20260619 15:07:43.697639 11515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:43.697940 11515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:43.698092 11515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:43.705737 11515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:43.705854 11515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:43.706029 11515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:07:43.715397 11515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:43.716769 11515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:43.718557 11515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:43.732254 11521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:43.732257 11523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:43.732779 11520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:43.733194 11515 server_base.cc:1034] running on GCE node
I20260619 15:07:43.733799 11515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:43.734921 11515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:43.736150 11515 hybrid_clock.cc:648] HybridClock initialized: now 1781881663736089 us; error 57 us; skew 500 ppm
I20260619 15:07:43.736487 11515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:07:43.738960 11515 webserver.cc:460] Webserver started at http://127.24.103.66:40193/ using document root <none> and password file <none>
I20260619 15:07:43.739686 11515 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:43.739822 11515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:43.740154 11515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:43.743202 11515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "5740a74e11c144949031ed759fa1f0e1"
format_stamp: "Formatted at 2026-06-19 15:07:43 on dist-test-slave-zpbw"
server_key: "d72dccf609ad91b40bf4403103274b71"
server_key_iv: "344ee1e167a7575ef7966d1784dd5177"
server_key_version: "encryptionkey@0"
I20260619 15:07:43.744024 11515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "5740a74e11c144949031ed759fa1f0e1"
format_stamp: "Formatted at 2026-06-19 15:07:43 on dist-test-slave-zpbw"
server_key: "d72dccf609ad91b40bf4403103274b71"
server_key_iv: "344ee1e167a7575ef7966d1784dd5177"
server_key_version: "encryptionkey@0"
I20260619 15:07:43.749557 11515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:07:43.753340 11529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:43.754926 11515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 15:07:43.755110 11515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "5740a74e11c144949031ed759fa1f0e1"
format_stamp: "Formatted at 2026-06-19 15:07:43 on dist-test-slave-zpbw"
server_key: "d72dccf609ad91b40bf4403103274b71"
server_key_iv: "344ee1e167a7575ef7966d1784dd5177"
server_key_version: "encryptionkey@0"
I20260619 15:07:43.755360 11515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:43.780006 11515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:43.781047 11515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:43.781402 11515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:43.782997 11515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:43.784963 11515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:43.785075 11515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:43.785185 11515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:43.785267 11515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:43.819646 11515 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:38093
I20260619 15:07:43.819741 11641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:38093 every 8 connection(s)
I20260619 15:07:43.820991 11515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:07:43.822583 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 11515
I20260619 15:07:43.822759 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:07:43.823343 24989 external_mini_cluster.cc:1428] Setting key fd07e6dc2387bb9e21de6a1b290d615b
I20260619 15:07:43.826434 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35263
--builtin_ntp_servers=127.24.103.84:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 15:07:43.834995 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:43.823729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:44153 (local address 127.24.103.126:35263)
0619 15:07:43.823946 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:43.823953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:43.825975 (+ 2022us) server_negotiation.cc:407] Connection header received
0619 15:07:43.828019 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:43.828025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:43.828122 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:43.828242 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:43.829398 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:43.830137 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:43.831875 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:43.832288 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:43.832433 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:43.833931 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:43.833940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:43.833948 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:43.833977 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:43.834021 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:43.834040 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:43.834131 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:43.834393 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 15:07:43.834515 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 15:07:43.836030 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:43.824366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35263 (local address 127.24.103.66:44153)
0619 15:07:43.825787 (+ 1421us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:43.825842 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:43.827726 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:43.828503 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:43.828512 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:43.828615 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:43.829198 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:43.829215 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:43.830306 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:43.830327 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:43.831712 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:43.831724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:43.832546 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:43.832553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:43.832799 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:43.833567 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:07:43.833570 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:43.833615 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:07:43.833706 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:43.834158 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:43.834166 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:43.834173 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 15:07:43.834346 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 15:07:43.834679 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":615,"threads_started":1}
I20260619 15:07:43.837625 11642 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35263
I20260619 15:07:43.837962 11642 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:43.838728 11642 heartbeater.cc:511] Master 127.24.103.126:35263 requested a full tablet report, sending...
I20260619 15:07:43.840252 11325 ts_manager.cc:194] Registered new tserver with Master: 5740a74e11c144949031ed759fa1f0e1 (127.24.103.66:38093)
I20260619 15:07:43.841084 11325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.66:44153
W20260619 15:07:44.165715 11645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:44.166064 11645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:44.166227 11645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:44.173866 11645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:44.174026 11645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:44.174202 11645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:07:44.183933 11645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:44.185344 11645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:44.187085 11645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:44.200657 11650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:44.200678 11651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:44.200660 11653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:44.204263 11645 server_base.cc:1034] running on GCE node
I20260619 15:07:44.204881 11645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:44.205806 11645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:44.207011 11645 hybrid_clock.cc:648] HybridClock initialized: now 1781881664206966 us; error 34 us; skew 500 ppm
I20260619 15:07:44.207350 11645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:07:44.209618 11645 webserver.cc:460] Webserver started at http://127.24.103.67:45775/ using document root <none> and password file <none>
I20260619 15:07:44.210301 11645 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:44.210445 11645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:44.210827 11645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:44.213724 11645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "58ef35890ef24c249db3128ad11b9101"
format_stamp: "Formatted at 2026-06-19 15:07:44 on dist-test-slave-zpbw"
server_key: "032b13fb6bfb23f4b8b9e52d39315f9d"
server_key_iv: "28e6fc7cba7f5403fd7a0f43337ca691"
server_key_version: "encryptionkey@0"
I20260619 15:07:44.214504 11645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "58ef35890ef24c249db3128ad11b9101"
format_stamp: "Formatted at 2026-06-19 15:07:44 on dist-test-slave-zpbw"
server_key: "032b13fb6bfb23f4b8b9e52d39315f9d"
server_key_iv: "28e6fc7cba7f5403fd7a0f43337ca691"
server_key_version: "encryptionkey@0"
I20260619 15:07:44.219729 11645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 15:07:44.223146 11659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:44.224489 11645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:07:44.224658 11645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "58ef35890ef24c249db3128ad11b9101"
format_stamp: "Formatted at 2026-06-19 15:07:44 on dist-test-slave-zpbw"
server_key: "032b13fb6bfb23f4b8b9e52d39315f9d"
server_key_iv: "28e6fc7cba7f5403fd7a0f43337ca691"
server_key_version: "encryptionkey@0"
I20260619 15:07:44.224874 11645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:44.250381 11645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:44.251358 11645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:44.251677 11645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:44.252786 11645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:44.254344 11645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:44.254433 11645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:44.254599 11645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:44.254680 11645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:44.288412 11645 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:46267
I20260619 15:07:44.288508 11771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:46267 every 8 connection(s)
I20260619 15:07:44.289759 11645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:07:44.293356 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 11645
I20260619 15:07:44.293565 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:07:44.294014 24989 external_mini_cluster.cc:1428] Setting key 290139d141d109de9293cf07131b75b7
I20260619 15:07:44.302927 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.292412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:38229 (local address 127.24.103.126:35263)
0619 15:07:44.292593 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:44.292597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:44.294201 (+ 1604us) server_negotiation.cc:407] Connection header received
0619 15:07:44.296611 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:44.296615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:44.296710 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:44.296801 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:44.297917 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.298632 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.300269 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.300666 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.300758 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.302093 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:44.302098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:44.302101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:44.302119 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:44.302150 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:44.302162 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:44.302231 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:44.302438 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 15:07:44.302516 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 15:07:44.303820 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.292965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35263 (local address 127.24.103.67:38229)
0619 15:07:44.294030 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:44.294079 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:44.296312 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:44.297074 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:44.297082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:44.297172 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:44.297770 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:44.297786 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.298756 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.298759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:44.300140 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:44.300152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.300843 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.300847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:44.301110 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.301769 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:07:44.301772 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:44.301817 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:07:44.301891 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:44.302246 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:44.302253 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:44.302259 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 15:07:44.302415 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 15:07:44.302697 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":350,"threads_started":1}
I20260619 15:07:44.305202 11772 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35263
I20260619 15:07:44.305510 11772 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:44.306205 11772 heartbeater.cc:511] Master 127.24.103.126:35263 requested a full tablet report, sending...
I20260619 15:07:44.307760 11325 ts_manager.cc:194] Registered new tserver with Master: 58ef35890ef24c249db3128ad11b9101 (127.24.103.67:46267)
I20260619 15:07:44.308876 11325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.67:38229
I20260619 15:07:44.309541 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:07:44.319602 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.313673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.24.103.126:35263)
0619 15:07:44.313860 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:44.313865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:44.314424 (+ 559us) server_negotiation.cc:407] Connection header received
0619 15:07:44.314565 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:44.314568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:44.314628 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:44.314714 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:44.315254 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.315956 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.317363 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.317770 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.317882 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.318980 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:44.318985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:44.318989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:44.319006 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:44.319040 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:44.319051 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:44.319161 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:44.319325 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 15:07:44.319373 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:07:44.323632 11325 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:07:44.330929 11325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:07:44.369863 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.343583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44803 (local address 127.0.0.1:51696)
0619 15:07:44.360436 (+ 16853us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:44.360464 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:44.360639 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:44.361989 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:44.361998 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:44.362040 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:44.362349 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:44.362359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.363875 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.363882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:44.366406 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:44.366422 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.368813 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.368822 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:44.369014 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.369065 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 15:07:44.369206 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 15:07:44.369410 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16646,"mutex_wait_us":1,"thread_start_us":18122,"threads_started":1}
I20260619 15:07:44.369927 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.343264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.24.103.65:44803)
0619 15:07:44.343822 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:44.343832 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:44.361100 (+ 17268us) server_negotiation.cc:407] Connection header received
0619 15:07:44.361222 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:44.361233 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:44.361540 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:44.361760 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:44.362525 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.363700 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.366649 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.368234 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.368366 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.368668 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:44.369281 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 15:07:44.369514 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":463,"threads_started":1}
I20260619 15:07:44.370689 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.342776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:46267 (local address 127.0.0.1:36890)
0619 15:07:44.358390 (+ 15614us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:44.358455 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:44.358709 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:44.362710 (+ 4001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:44.362717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:44.362747 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:44.363155 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:44.363166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.365399 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.365405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:44.367523 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:44.367535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.369714 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.369720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:44.369988 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.370032 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 15:07:44.370113 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 15:07:44.370260 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15210,"mutex_wait_us":63,"thread_start_us":16206,"threads_started":1}
I20260619 15:07:44.370829 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.344122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:38093 (local address 127.0.0.1:40138)
0619 15:07:44.361247 (+ 17125us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:44.361269 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:44.361391 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:44.363135 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:44.363139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:44.363161 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:44.363550 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:44.363566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.365463 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.365467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:44.367728 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:44.367743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.369875 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.369885 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:44.370142 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.370173 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 15:07:44.370255 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 15:07:44.370402 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16973,"mutex_wait_us":127,"thread_start_us":15948,"threads_started":1}
I20260619 15:07:44.370986 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.344392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.24.103.66:38093)
0619 15:07:44.362353 (+ 17961us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:44.362361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:44.362387 (+ 26us) server_negotiation.cc:407] Connection header received
0619 15:07:44.362480 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:44.362487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:44.362696 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:44.362899 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:44.363757 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.365258 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.367949 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.369389 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.369507 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.369727 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:44.370309 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 15:07:44.370513 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17797,"thread_start_us":18095,"threads_started":1}
I20260619 15:07:44.372373 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.343118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.24.103.67:46267)
0619 15:07:44.362074 (+ 18956us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:44.362085 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:44.362125 (+ 40us) server_negotiation.cc:407] Connection header received
0619 15:07:44.362244 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:44.362252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:44.362492 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:44.363394 (+ 902us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:44.363698 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.365258 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.367815 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.369149 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.369306 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.369576 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:44.370965 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0619 15:07:44.371912 (+ 947us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18706,"thread_start_us":18544,"threads_started":1}
I20260619 15:07:44.375016 11511 heartbeater.cc:503] Master 127.24.103.126:35263 was elected leader, sending a full tablet report...
I20260619 15:07:44.374737 11577 tablet_service.cc:1467] Processing CreateTablet for tablet 8db99208b15642469b79388b6714138c (DEFAULT_TABLE table=test-table [id=4b06c785d2ff4fdda4244e6a534c47f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:44.374809 11446 tablet_service.cc:1467] Processing CreateTablet for tablet 8db99208b15642469b79388b6714138c (DEFAULT_TABLE table=test-table [id=4b06c785d2ff4fdda4244e6a534c47f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:44.376268 11577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db99208b15642469b79388b6714138c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:44.376263 11446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db99208b15642469b79388b6714138c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:44.376046 11707 tablet_service.cc:1467] Processing CreateTablet for tablet 8db99208b15642469b79388b6714138c (DEFAULT_TABLE table=test-table [id=4b06c785d2ff4fdda4244e6a534c47f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:44.377485 11707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db99208b15642469b79388b6714138c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:44.388415 11790 tablet_bootstrap.cc:492] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd: Bootstrap starting.
I20260619 15:07:44.391149 11791 tablet_bootstrap.cc:492] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1: Bootstrap starting.
I20260619 15:07:44.393419 11790 tablet_bootstrap.cc:654] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:44.394872 11790 log.cc:824] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:44.395048 11791 tablet_bootstrap.cc:654] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:44.396363 11791 log.cc:824] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:44.398392 11790 tablet_bootstrap.cc:492] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd: No bootstrap required, opened a new log
I20260619 15:07:44.398701 11790 ts_tablet_manager.cc:1397] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 15:07:44.399591 11791 tablet_bootstrap.cc:492] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1: No bootstrap required, opened a new log
I20260619 15:07:44.399860 11791 ts_tablet_manager.cc:1397] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 15:07:44.406898 11792 tablet_bootstrap.cc:492] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101: Bootstrap starting.
I20260619 15:07:44.409581 11790 raft_consensus.cc:348] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ef35890ef24c249db3128ad11b9101" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46267 } } peers { permanent_uuid: "260781d0e0cd4f12bc6e16528554e4dd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44803 } } peers { permanent_uuid: "5740a74e11c144949031ed759fa1f0e1" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38093 } }
I20260619 15:07:44.410130 11790 raft_consensus.cc:374] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:44.410280 11790 raft_consensus.cc:729] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 260781d0e0cd4f12bc6e16528554e4dd, State: Initialized, Role: FOLLOWER
I20260619 15:07:44.410960 11792 tablet_bootstrap.cc:654] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:44.411029 11790 consensus_queue.cc:260] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ef35890ef24c249db3128ad11b9101" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46267 } } peers { permanent_uuid: "260781d0e0cd4f12bc6e16528554e4dd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44803 } } peers { permanent_uuid: "5740a74e11c144949031ed759fa1f0e1" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38093 } }
I20260619 15:07:44.412544 11792 log.cc:824] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:44.413674 11791 raft_consensus.cc:348] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ef35890ef24c249db3128ad11b9101" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46267 } } peers { permanent_uuid: "260781d0e0cd4f12bc6e16528554e4dd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44803 } } peers { permanent_uuid: "5740a74e11c144949031ed759fa1f0e1" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38093 } }
I20260619 15:07:44.414176 11791 raft_consensus.cc:374] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:44.414299 11791 raft_consensus.cc:729] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5740a74e11c144949031ed759fa1f0e1, State: Initialized, Role: FOLLOWER
I20260619 15:07:44.415004 11791 consensus_queue.cc:260] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ef35890ef24c249db3128ad11b9101" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46267 } } peers { permanent_uuid: "260781d0e0cd4f12bc6e16528554e4dd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44803 } } peers { permanent_uuid: "5740a74e11c144949031ed759fa1f0e1" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38093 } }
I20260619 15:07:44.416338 11642 heartbeater.cc:503] Master 127.24.103.126:35263 was elected leader, sending a full tablet report...
I20260619 15:07:44.416860 11791 ts_tablet_manager.cc:1428] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260619 15:07:44.418097 11790 ts_tablet_manager.cc:1428] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd: Time spent starting tablet: real 0.019s user 0.011s sys 0.004s
I20260619 15:07:44.421422 11792 tablet_bootstrap.cc:492] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101: No bootstrap required, opened a new log
I20260619 15:07:44.421731 11792 ts_tablet_manager.cc:1397] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.009s
I20260619 15:07:44.429306 11792 raft_consensus.cc:348] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ef35890ef24c249db3128ad11b9101" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46267 } } peers { permanent_uuid: "260781d0e0cd4f12bc6e16528554e4dd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44803 } } peers { permanent_uuid: "5740a74e11c144949031ed759fa1f0e1" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38093 } }
I20260619 15:07:44.429706 11792 raft_consensus.cc:374] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:44.429837 11792 raft_consensus.cc:729] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58ef35890ef24c249db3128ad11b9101, State: Initialized, Role: FOLLOWER
I20260619 15:07:44.430429 11792 consensus_queue.cc:260] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ef35890ef24c249db3128ad11b9101" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46267 } } peers { permanent_uuid: "260781d0e0cd4f12bc6e16528554e4dd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44803 } } peers { permanent_uuid: "5740a74e11c144949031ed759fa1f0e1" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38093 } }
I20260619 15:07:44.431485 11772 heartbeater.cc:503] Master 127.24.103.126:35263 was elected leader, sending a full tablet report...
I20260619 15:07:44.432211 11792 ts_tablet_manager.cc:1428] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
W20260619 15:07:44.541626 11773 tablet.cc:2369] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:07:44.563835 11797 raft_consensus.cc:482] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:07:44.564148 11797 raft_consensus.cc:504] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ef35890ef24c249db3128ad11b9101" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46267 } } peers { permanent_uuid: "260781d0e0cd4f12bc6e16528554e4dd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44803 } } peers { permanent_uuid: "5740a74e11c144949031ed759fa1f0e1" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38093 } }
I20260619 15:07:44.565805 11797 leader_election.cc:288] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58ef35890ef24c249db3128ad11b9101 (127.24.103.67:46267), 260781d0e0cd4f12bc6e16528554e4dd (127.24.103.65:44803)
I20260619 15:07:44.572892 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.566432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:55495 (local address 127.24.103.65:44803)
0619 15:07:44.566700 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:44.566707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:44.567320 (+ 613us) server_negotiation.cc:407] Connection header received
0619 15:07:44.567515 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:44.567519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:44.567618 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:44.567709 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:44.568340 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.569013 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.571112 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.571898 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.572055 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.572137 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:44.572497 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 15:07:44.572627 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 15:07:44.572979 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.566152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:37405 (local address 127.24.103.67:46267)
0619 15:07:44.566363 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:44.566371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:44.567245 (+ 874us) server_negotiation.cc:407] Connection header received
0619 15:07:44.567446 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:44.567453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:44.567535 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:44.567603 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:44.568248 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.568996 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.571163 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.571927 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.572114 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.572185 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:44.572515 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 15:07:44.572645 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:07:44.573114 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.566328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44803 (local address 127.24.103.66:55495)
0619 15:07:44.567163 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:44.567179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:44.567367 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:44.567757 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:44.567761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:44.567799 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:44.568174 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:44.568188 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.569173 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.569177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:44.570924 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:44.570940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.572098 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.572105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:44.572290 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.572340 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 15:07:44.572447 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 15:07:44.572609 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":109,"thread_start_us":548,"threads_started":1}
W20260619 15:07:44.573232 11643 tablet.cc:2369] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:07:44.573593 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.566205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:46267 (local address 127.24.103.66:37405)
0619 15:07:44.567097 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:44.567120 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:44.567288 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:44.567652 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:44.567660 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:44.567687 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:44.568092 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:44.568101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.569155 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.569160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:44.570951 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:44.570962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.572139 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.572145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:44.572325 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.572348 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 15:07:44.572451 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 15:07:44.572612 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":118,"thread_start_us":468,"threads_started":1}
I20260619 15:07:44.574440 11466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db99208b15642469b79388b6714138c" candidate_uuid: "5740a74e11c144949031ed759fa1f0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "260781d0e0cd4f12bc6e16528554e4dd" is_pre_election: true
I20260619 15:07:44.574617 11727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db99208b15642469b79388b6714138c" candidate_uuid: "5740a74e11c144949031ed759fa1f0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58ef35890ef24c249db3128ad11b9101" is_pre_election: true
I20260619 15:07:44.575081 11466 raft_consensus.cc:2454] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5740a74e11c144949031ed759fa1f0e1 in term 0.
I20260619 15:07:44.575171 11727 raft_consensus.cc:2454] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5740a74e11c144949031ed759fa1f0e1 in term 0.
I20260619 15:07:44.575807 11531 leader_election.cc:302] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 260781d0e0cd4f12bc6e16528554e4dd, 5740a74e11c144949031ed759fa1f0e1; no voters:
I20260619 15:07:44.576231 11797 raft_consensus.cc:2781] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:07:44.576377 11797 raft_consensus.cc:482] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:07:44.576476 11797 raft_consensus.cc:3037] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:44.578692 11797 raft_consensus.cc:504] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ef35890ef24c249db3128ad11b9101" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46267 } } peers { permanent_uuid: "260781d0e0cd4f12bc6e16528554e4dd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44803 } } peers { permanent_uuid: "5740a74e11c144949031ed759fa1f0e1" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38093 } }
I20260619 15:07:44.579437 11797 leader_election.cc:288] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [CANDIDATE]: Term 1 election: Requested vote from peers 58ef35890ef24c249db3128ad11b9101 (127.24.103.67:46267), 260781d0e0cd4f12bc6e16528554e4dd (127.24.103.65:44803)
I20260619 15:07:44.579923 11727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db99208b15642469b79388b6714138c" candidate_uuid: "5740a74e11c144949031ed759fa1f0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58ef35890ef24c249db3128ad11b9101"
I20260619 15:07:44.579934 11466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db99208b15642469b79388b6714138c" candidate_uuid: "5740a74e11c144949031ed759fa1f0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "260781d0e0cd4f12bc6e16528554e4dd"
I20260619 15:07:44.580133 11727 raft_consensus.cc:3037] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:44.580149 11466 raft_consensus.cc:3037] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:44.582340 11727 raft_consensus.cc:2454] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5740a74e11c144949031ed759fa1f0e1 in term 1.
I20260619 15:07:44.582409 11466 raft_consensus.cc:2454] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5740a74e11c144949031ed759fa1f0e1 in term 1.
I20260619 15:07:44.582855 11533 leader_election.cc:302] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5740a74e11c144949031ed759fa1f0e1, 58ef35890ef24c249db3128ad11b9101; no voters:
I20260619 15:07:44.583225 11797 raft_consensus.cc:2781] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:44.583693 11797 raft_consensus.cc:686] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 1 LEADER]: Becoming Leader. State: Replica: 5740a74e11c144949031ed759fa1f0e1, State: Running, Role: LEADER
I20260619 15:07:44.584229 11797 consensus_queue.cc:237] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ef35890ef24c249db3128ad11b9101" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46267 } } peers { permanent_uuid: "260781d0e0cd4f12bc6e16528554e4dd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44803 } } peers { permanent_uuid: "5740a74e11c144949031ed759fa1f0e1" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38093 } }
I20260619 15:07:44.589177 11325 catalog_manager.cc:5495] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5740a74e11c144949031ed759fa1f0e1 (127.24.103.66). New cstate: current_term: 1 leader_uuid: "5740a74e11c144949031ed759fa1f0e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ef35890ef24c249db3128ad11b9101" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "260781d0e0cd4f12bc6e16528554e4dd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5740a74e11c144949031ed759fa1f0e1" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 38093 } health_report { overall_health: HEALTHY } } }
W20260619 15:07:44.599504 11512 tablet.cc:2369] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:07:44.612855 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.607592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.24.103.66:38093)
0619 15:07:44.607761 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:44.607765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:44.607793 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:07:44.607883 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:44.607886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:44.607971 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:44.608101 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:07:44.608443 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.609210 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.610538 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.611103 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.611197 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.611737 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:07:44.612332 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:07:44.612462 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:44.612599 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 15:07:44.612635 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 15:07:44.621061 11466 raft_consensus.cc:1261] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd [term 1 FOLLOWER]: Refusing update from remote peer 5740a74e11c144949031ed759fa1f0e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:07:44.621058 11727 raft_consensus.cc:1261] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101 [term 1 FOLLOWER]: Refusing update from remote peer 5740a74e11c144949031ed759fa1f0e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:07:44.622089 11801 consensus_queue.cc:1035] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "260781d0e0cd4f12bc6e16528554e4dd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 44803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:07:44.622409 11797 consensus_queue.cc:1035] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58ef35890ef24c249db3128ad11b9101" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:07:44.641353 11806 mvcc.cc:205] Tried to move back new op lower bound from 7298587298278125568 to 7298587298141241344. Current Snapshot: MvccSnapshot[applied={T|T < 7298587298278125568}]
I20260619 15:07:44.652243 11325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:07:44.652534 11325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:07:44.656206 11325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e808529a7c341d4a4142d2b5f4dfa67: Sending DeleteTablet for 3 replicas of tablet 8db99208b15642469b79388b6714138c
I20260619 15:07:44.657430 11707 tablet_service.cc:1514] Processing DeleteTablet for tablet 8db99208b15642469b79388b6714138c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:44 UTC) from {username='slave'} at 127.0.0.1:36890
I20260619 15:07:44.657475 11446 tablet_service.cc:1514] Processing DeleteTablet for tablet 8db99208b15642469b79388b6714138c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:44 UTC) from {username='slave'} at 127.0.0.1:51696
I20260619 15:07:44.658072 11577 tablet_service.cc:1514] Processing DeleteTablet for tablet 8db99208b15642469b79388b6714138c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:44 UTC) from {username='slave'} at 127.0.0.1:40138
I20260619 15:07:44.659411 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 11384
I20260619 15:07:44.668668 11814 tablet_replica.cc:330] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101: stopping tablet replica
I20260619 15:07:44.669329 11814 raft_consensus.cc:2229] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:07:44.669912 11814 raft_consensus.cc:2258] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:44.670866 11815 tablet_replica.cc:330] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd: stopping tablet replica
I20260619 15:07:44.671939 11816 tablet_replica.cc:330] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1: stopping tablet replica
I20260619 15:07:44.672468 11815 raft_consensus.cc:2229] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:07:44.672964 11814 ts_tablet_manager.cc:1905] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:44.673133 11816 raft_consensus.cc:2229] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:07:44.673915 11816 raft_consensus.cc:2258] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:44.674489 11815 raft_consensus.cc:2258] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:44.677105 11816 ts_tablet_manager.cc:1905] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:44.684307 11815 ts_tablet_manager.cc:1905] T 8db99208b15642469b79388b6714138c P 260781d0e0cd4f12bc6e16528554e4dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:44.685093 11816 ts_tablet_manager.cc:1918] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:44.685370 11816 log.cc:1196] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/8db99208b15642469b79388b6714138c
I20260619 15:07:44.685536 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.661415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.24.103.65:44803)
0619 15:07:44.661652 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:44.661661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:44.676978 (+ 15317us) server_negotiation.cc:407] Connection header received
0619 15:07:44.677213 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:44.677219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:44.677332 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:44.677437 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:44.677907 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.679104 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.681175 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.681817 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.681916 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.683649 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:44.683661 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:44.683680 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:44.684517 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:44.684583 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:44.684625 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:44.684713 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:44.685017 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 15:07:44.685145 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:07:44.686285 11816 ts_tablet_manager.cc:1939] T 8db99208b15642469b79388b6714138c P 5740a74e11c144949031ed759fa1f0e1: Deleting consensus metadata
I20260619 15:07:44.686362 11814 ts_tablet_manager.cc:1918] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:44.686553 11814 log.cc:1196] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/8db99208b15642469b79388b6714138c
I20260619 15:07:44.686843 11506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:44.687158 11814 ts_tablet_manager.cc:1939] T 8db99208b15642469b79388b6714138c P 58ef35890ef24c249db3128ad11b9101: Deleting consensus metadata
I20260619 15:07:44.689018 11312 catalog_manager.cc:4841] TS 58ef35890ef24c249db3128ad11b9101 (127.24.103.67:46267): tablet 8db99208b15642469b79388b6714138c (table test-table [id=4b06c785d2ff4fdda4244e6a534c47f4]) successfully deleted
I20260619 15:07:44.689081 11311 catalog_manager.cc:4841] TS 5740a74e11c144949031ed759fa1f0e1 (127.24.103.66:38093): tablet 8db99208b15642469b79388b6714138c (table test-table [id=4b06c785d2ff4fdda4244e6a534c47f4]) successfully deleted
I20260619 15:07:44.781700 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 11384
I20260619 15:07:44.802613 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 11515
W20260619 15:07:44.802665 11310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.103.65:44803 (error 108)
I20260619 15:07:44.804693 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.804153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44803 (local address 127.0.0.1:51720)
0619 15:07:44.804371 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:44.804509 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 15:07:44.805202 11310 catalog_manager.cc:4584] TS 260781d0e0cd4f12bc6e16528554e4dd (127.24.103.65:44803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8db99208b15642469b79388b6714138c: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44803: connect: Connection refused (error 111)
I20260619 15:07:44.811026 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.803917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.24.103.66:38093)
0619 15:07:44.804099 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:44.804103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:44.804339 (+ 236us) server_negotiation.cc:407] Connection header received
0619 15:07:44.804628 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:44.804634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:44.804739 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:44.804861 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:44.805348 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.806228 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.807673 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.808106 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.808292 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.809454 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:44.809463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:44.809481 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:44.810124 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:44.810180 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:44.810201 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:44.810363 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:44.810503 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 15:07:44.810615 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:07:44.811818 11637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:44.859030 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.858346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44803 (local address 127.0.0.1:51732)
0619 15:07:44.858625 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:44.858755 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 15:07:44.860199 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.859785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44803 (local address 127.0.0.1:51748)
0619 15:07:44.859951 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:44.860039 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 15:07:44.921056 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.920475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44803 (local address 127.0.0.1:51750)
0619 15:07:44.920735 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:44.920864 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 15:07:44.922180 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.921780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44803 (local address 127.0.0.1:51766)
0619 15:07:44.921943 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:44.922046 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 15:07:44.935344 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 11515
I20260619 15:07:44.954069 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 11645
I20260619 15:07:44.961320 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:44.954941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.24.103.67:46267)
0619 15:07:44.955125 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:44.955132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:44.955189 (+ 57us) server_negotiation.cc:407] Connection header received
0619 15:07:44.955425 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:44.955435 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:44.955520 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:44.955637 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:44.956037 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.956736 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.958060 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:44.958473 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:44.958611 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:44.959933 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:44.959942 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:44.959957 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:44.960515 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:44.960578 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:44.960598 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:44.960723 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:44.960875 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 15:07:44.960987 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 15:07:44.961972 11767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:45.002743 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:45.002237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44803 (local address 127.0.0.1:51770)
0619 15:07:45.002469 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:45.002563 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 15:07:45.003928 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:45.003536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44803 (local address 127.0.0.1:51774)
0619 15:07:45.003736 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:45.003820 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 15:07:45.053476 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 11645
I20260619 15:07:45.071748 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 11294
I20260619 15:07:45.073027 11355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:45.184836 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:45.184136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:44803 (local address 127.0.0.1:51780)
0619 15:07:45.184465 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:45.184583 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260619 15:07:45.184890 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 11294
2026-06-19T15:07:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2953 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:07:45 dist-test-slave-zpbw krb5kdc[11825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:07:45 dist-test-slave-zpbw krb5kdc[11825](info): set up 2 sockets
Jun 19 15:07:45 dist-test-slave-zpbw krb5kdc[11825](info): commencing operation
krb5kdc: starting...
W20260619 15:07:46.299340 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:07:46 dist-test-slave-zpbw krb5kdc[11825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:07:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:07:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:07:46.558761 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38169
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:46.909175 11838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:46.909503 11838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:46.909689 11838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:46.917330 11838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:07:46.917465 11838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:46.917583 11838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:46.917713 11838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:07:46.917799 11838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:07:46.928340 11838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:46.929888 11838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:46.931367 11838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:46.943720 11844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:46.943771 11846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:46.943795 11843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:46.945070 11838 server_base.cc:1034] running on GCE node
I20260619 15:07:46.945920 11838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:46.947489 11838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:46.948838 11838 hybrid_clock.cc:648] HybridClock initialized: now 1781881666948788 us; error 58 us; skew 500 ppm
I20260619 15:07:46.949295 11838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:46 dist-test-slave-zpbw krb5kdc[11825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881666, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:46.952623 11838 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:07:46.954411 11838 webserver.cc:460] Webserver started at http://127.24.103.126:33167/ using document root <none> and password file <none>
I20260619 15:07:46.955258 11838 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:46.955423 11838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:46.955749 11838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:46.959122 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "966dc5a7d66846e8b764105b8ac10978"
format_stamp: "Formatted at 2026-06-19 15:07:46 on dist-test-slave-zpbw"
server_key: "610d2614769cfc194b8e9be6a77659e6"
server_key_iv: "4b91fbc9b0b1fa2ac32ced89dee7707c"
server_key_version: "encryptionkey@0"
I20260619 15:07:46.959954 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "966dc5a7d66846e8b764105b8ac10978"
format_stamp: "Formatted at 2026-06-19 15:07:46 on dist-test-slave-zpbw"
server_key: "610d2614769cfc194b8e9be6a77659e6"
server_key_iv: "4b91fbc9b0b1fa2ac32ced89dee7707c"
server_key_version: "encryptionkey@0"
I20260619 15:07:46.965544 11838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:07:46.969259 11853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:46.970657 11838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 15:07:46.970913 11838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "966dc5a7d66846e8b764105b8ac10978"
format_stamp: "Formatted at 2026-06-19 15:07:46 on dist-test-slave-zpbw"
server_key: "610d2614769cfc194b8e9be6a77659e6"
server_key_iv: "4b91fbc9b0b1fa2ac32ced89dee7707c"
server_key_version: "encryptionkey@0"
I20260619 15:07:46.971145 11838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:46.987507 11838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:46.991925 11838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:46.992273 11838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:47.012225 11838 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:38169
I20260619 15:07:47.012231 11904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:38169 every 8 connection(s)
I20260619 15:07:47.013813 11838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:07:47.014967 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 11838
I20260619 15:07:47.015208 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:07:47.015755 24989 external_mini_cluster.cc:1428] Setting key 4b270c3e5cb6d63361a4b1cc8d5c73cc
I20260619 15:07:47.018761 11905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 15:07:47 dist-test-slave-zpbw krb5kdc[11825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:07:47.033444 11905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978: Bootstrap starting.
I20260619 15:07:47.038110 11905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:47.039415 11905 log.cc:824] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:47.043298 11905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978: No bootstrap required, opened a new log
I20260619 15:07:47.043723 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:47.018101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.24.103.126:38169)
0619 15:07:47.019243 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:47.019260 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:47.019704 (+ 444us) server_negotiation.cc:407] Connection header received
0619 15:07:47.022116 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:47.022150 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:47.022767 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:47.023291 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:47.024819 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:47.026954 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:47.028321 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:47.029421 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:47.029625 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:47.034237 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:47.034250 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:47.034272 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:47.034329 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:47.038596 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:47.039527 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:47.039535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:47.039544 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:47.039791 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:47.040293 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:47.040297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:47.040299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:47.040954 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:47.041369 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:47.041803 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 15:07:47.042276 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":485,"threads_started":1}
I20260619 15:07:47.051977 11905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966dc5a7d66846e8b764105b8ac10978" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38169 } }
I20260619 15:07:47.052387 11905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:47.052470 11905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966dc5a7d66846e8b764105b8ac10978, State: Initialized, Role: FOLLOWER
I20260619 15:07:47.053067 11905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966dc5a7d66846e8b764105b8ac10978" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38169 } }
I20260619 15:07:47.053376 11905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:07:47.053488 11905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:07:47.053633 11905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:47.056036 11905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966dc5a7d66846e8b764105b8ac10978" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38169 } }
I20260619 15:07:47.056536 11905 leader_election.cc:302] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 966dc5a7d66846e8b764105b8ac10978; no voters:
I20260619 15:07:47.057077 11905 leader_election.cc:288] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:07:47.057240 11910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:47.057984 11910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [term 1 LEADER]: Becoming Leader. State: Replica: 966dc5a7d66846e8b764105b8ac10978, State: Running, Role: LEADER
I20260619 15:07:47.058511 11910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966dc5a7d66846e8b764105b8ac10978" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38169 } }
I20260619 15:07:47.059332 11905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:07:47.061595 11912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 966dc5a7d66846e8b764105b8ac10978. Latest consensus state: current_term: 1 leader_uuid: "966dc5a7d66846e8b764105b8ac10978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966dc5a7d66846e8b764105b8ac10978" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38169 } } }
I20260619 15:07:47.062090 11912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:47.063062 11911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "966dc5a7d66846e8b764105b8ac10978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966dc5a7d66846e8b764105b8ac10978" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38169 } } }
I20260619 15:07:47.063437 11911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:47.066133 11917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:07:47.072077 11917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:07:47.079294 11927 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 15:07:47.084025 11917 catalog_manager.cc:1347] Generated new cluster ID: d06c461c8a9048169865f2aeecce1694
I20260619 15:07:47.084164 11917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:07:47.091414 11917 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:07:47.093031 11917 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:07:47.104606 11917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978: Generated new TSK 0
I20260619 15:07:47.105553 11917 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260619 15:07:47.174001 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38169
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:47.523025 11931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:47.523407 11931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:47.523568 11931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:47.531275 11931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:47.531396 11931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:47.531567 11931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:07:47.541944 11931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:47.543455 11931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:47.544909 11931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:47.559396 11936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:47.559423 11937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:47.559396 11939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:47.561620 11931 server_base.cc:1034] running on GCE node
I20260619 15:07:47.562309 11931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:47.563539 11931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:47.564812 11931 hybrid_clock.cc:648] HybridClock initialized: now 1781881667564725 us; error 75 us; skew 500 ppm
I20260619 15:07:47.565140 11931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:47 dist-test-slave-zpbw krb5kdc[11825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881667, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:47.568543 11931 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260619 15:07:47.570144 11931 webserver.cc:460] Webserver started at http://127.24.103.65:36071/ using document root <none> and password file <none>
I20260619 15:07:47.570919 11931 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:47.571024 11931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:47.571295 11931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:47.574348 11931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "3f06eff8dee84455945bde563878d9d0"
format_stamp: "Formatted at 2026-06-19 15:07:47 on dist-test-slave-zpbw"
server_key: "430dafd10bb7ddfdd238e829256ead1d"
server_key_iv: "375a1216978a5c3d726262b2d99f5200"
server_key_version: "encryptionkey@0"
I20260619 15:07:47.575258 11931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "3f06eff8dee84455945bde563878d9d0"
format_stamp: "Formatted at 2026-06-19 15:07:47 on dist-test-slave-zpbw"
server_key: "430dafd10bb7ddfdd238e829256ead1d"
server_key_iv: "375a1216978a5c3d726262b2d99f5200"
server_key_version: "encryptionkey@0"
I20260619 15:07:47.581091 11931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:07:47.584914 11946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:47.586407 11931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:07:47.586611 11931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "3f06eff8dee84455945bde563878d9d0"
format_stamp: "Formatted at 2026-06-19 15:07:47 on dist-test-slave-zpbw"
server_key: "430dafd10bb7ddfdd238e829256ead1d"
server_key_iv: "375a1216978a5c3d726262b2d99f5200"
server_key_version: "encryptionkey@0"
I20260619 15:07:47.586903 11931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:47.609678 11931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:47.614041 11931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:47.614364 11931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:47.615500 11931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:47.616946 11931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:47.617022 11931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:47.617195 11931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:47.617281 11931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:47.652338 11931 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:39813
I20260619 15:07:47.652381 12058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:39813 every 8 connection(s)
I20260619 15:07:47.653753 11931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:07:47.661053 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 11931
I20260619 15:07:47.661255 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:07:47.661813 24989 external_mini_cluster.cc:1428] Setting key 692785fb219df7d7f812c2030f448737
Jun 19 15:07:47 dist-test-slave-zpbw krb5kdc[11825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881667, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:07:47.677394 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:47.656598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:49129 (local address 127.24.103.126:38169)
0619 15:07:47.657099 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:47.657103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:47.657984 (+ 881us) server_negotiation.cc:407] Connection header received
0619 15:07:47.660005 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:47.660011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:47.660107 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:47.660230 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:47.662373 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:47.663397 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:47.665840 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:47.666410 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:47.666518 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:47.671050 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:47.671055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:47.671059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:47.671095 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:47.673750 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:47.674633 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:47.674638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:47.674640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:47.674817 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:47.675351 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:47.675357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:47.675360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:47.675710 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:47.675873 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:47.676753 (+ 880us) server_negotiation.cc:299] Negotiation successful
0619 15:07:47.677003 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":281,"threads_started":1}
I20260619 15:07:47.678440 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:47.657086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38169 (local address 127.24.103.65:49129)
0619 15:07:47.657807 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:47.657850 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:47.659709 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:47.660482 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:47.660491 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:47.661285 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:47.662137 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:47.662163 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:47.663580 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:47.663587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:47.665668 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:47.665690 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:47.666674 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:47.666684 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:47.667091 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:47.667870 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:07:47.667912 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:47.670719 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:47.673930 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:47.673940 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:47.673960 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:47.674485 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:47.674958 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:47.674963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:47.674967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:47.675226 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:47.675845 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:47.675854 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:47.676341 (+ 487us) client_negotiation.cc:769] Sending connection context
0619 15:07:47.676629 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 15:07:47.677215 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":282,"threads_started":1}
I20260619 15:07:47.680159 12059 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38169
I20260619 15:07:47.680548 12059 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:47.681511 12059 heartbeater.cc:511] Master 127.24.103.126:38169 requested a full tablet report, sending...
I20260619 15:07:47.684314 11870 ts_manager.cc:194] Registered new tserver with Master: 3f06eff8dee84455945bde563878d9d0 (127.24.103.65:39813)
I20260619 15:07:47.686738 11870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:49129
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260619 15:07:47.721521 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38169
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:48.066009 12065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:48.066301 12065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:48.066396 12065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:48.074065 12065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:48.074218 12065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:48.074340 12065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:07:48.084774 12065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:48.086189 12065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:48.087656 12065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:48.101893 12070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:48.101825 12073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:48.101874 12071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:48.103037 12065 server_base.cc:1034] running on GCE node
I20260619 15:07:48.103746 12065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:48.104874 12065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:48.106122 12065 hybrid_clock.cc:648] HybridClock initialized: now 1781881668106076 us; error 44 us; skew 500 ppm
I20260619 15:07:48.106477 12065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:48 dist-test-slave-zpbw krb5kdc[11825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881668, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:48.109791 12065 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260619 15:07:48.111594 12065 webserver.cc:460] Webserver started at http://127.24.103.66:46869/ using document root <none> and password file <none>
I20260619 15:07:48.112411 12065 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:48.112579 12065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:48.112931 12065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:48.116407 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "1361884e9b5b48a29c56fb77e3dc2e05"
format_stamp: "Formatted at 2026-06-19 15:07:48 on dist-test-slave-zpbw"
server_key: "e2f117ccab4f66fe69e1652a38410179"
server_key_iv: "a56c178f9639ee6c6880fb1238d34293"
server_key_version: "encryptionkey@0"
I20260619 15:07:48.117456 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "1361884e9b5b48a29c56fb77e3dc2e05"
format_stamp: "Formatted at 2026-06-19 15:07:48 on dist-test-slave-zpbw"
server_key: "e2f117ccab4f66fe69e1652a38410179"
server_key_iv: "a56c178f9639ee6c6880fb1238d34293"
server_key_version: "encryptionkey@0"
I20260619 15:07:48.123185 12065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:07:48.127343 12080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:48.129069 12065 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 15:07:48.129251 12065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "1361884e9b5b48a29c56fb77e3dc2e05"
format_stamp: "Formatted at 2026-06-19 15:07:48 on dist-test-slave-zpbw"
server_key: "e2f117ccab4f66fe69e1652a38410179"
server_key_iv: "a56c178f9639ee6c6880fb1238d34293"
server_key_version: "encryptionkey@0"
I20260619 15:07:48.129464 12065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:48.152235 12065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:48.156886 12065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:48.157140 12065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:48.158133 12065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:48.159601 12065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:48.159679 12065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:48.159812 12065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:48.159883 12065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:48.194401 12065 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:42349
I20260619 15:07:48.194484 12192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:42349 every 8 connection(s)
I20260619 15:07:48.195935 12065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:07:48.198275 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 12065
I20260619 15:07:48.198490 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:07:48.199090 24989 external_mini_cluster.cc:1428] Setting key c8db3de681654cd443cb4f00126b2b53
Jun 19 15:07:48 dist-test-slave-zpbw krb5kdc[11825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881668, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:07:48.219874 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.198553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:56671 (local address 127.24.103.126:38169)
0619 15:07:48.199069 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:48.199076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:48.200210 (+ 1134us) server_negotiation.cc:407] Connection header received
0619 15:07:48.202567 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:48.202572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:48.202638 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:48.202729 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:48.205056 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.206137 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.208831 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.209527 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.209661 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.213694 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:48.213700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:48.213703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:48.213739 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:48.216499 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:48.217339 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:48.217344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:48.217346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:48.217476 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:48.217956 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:48.217960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:48.217962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:48.218241 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:48.218414 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:48.219214 (+ 800us) server_negotiation.cc:299] Negotiation successful
0619 15:07:48.219494 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
I20260619 15:07:48.220929 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.199293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38169 (local address 127.24.103.66:56671)
0619 15:07:48.200043 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:48.200088 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:48.202240 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:48.203072 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:48.203087 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:48.203958 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:48.204855 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.204882 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.206315 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.206323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.208643 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.208666 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.209808 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.209818 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.210184 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.211067 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:07:48.211095 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:48.213394 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:48.216670 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:48.216680 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:48.216701 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:48.217178 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:48.217599 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:48.217607 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:48.217610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:48.217822 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:48.218384 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:48.218393 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:48.218824 (+ 431us) client_negotiation.cc:769] Sending connection context
0619 15:07:48.219118 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 15:07:48.219723 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":318,"threads_started":1}
I20260619 15:07:48.222450 12193 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38169
I20260619 15:07:48.222815 12193 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:48.223579 12193 heartbeater.cc:511] Master 127.24.103.126:38169 requested a full tablet report, sending...
I20260619 15:07:48.225291 11870 ts_manager.cc:194] Registered new tserver with Master: 1361884e9b5b48a29c56fb77e3dc2e05 (127.24.103.66:42349)
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260619 15:07:48.226326 11870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:56671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260619 15:07:48.258059 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38169
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:48.604144 12199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:48.604463 12199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:48.604564 12199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:48.612087 12199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:48.612264 12199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:48.612455 12199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:07:48.622875 12199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:48.624341 12199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:48.625653 12199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:48.639254 12207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:48.639209 12204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:48.639775 12205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:48.640228 12199 server_base.cc:1034] running on GCE node
I20260619 15:07:48.640889 12199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:48.641896 12199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:48.643092 12199 hybrid_clock.cc:648] HybridClock initialized: now 1781881668643020 us; error 66 us; skew 500 ppm
I20260619 15:07:48.643428 12199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:48 dist-test-slave-zpbw krb5kdc[11825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881668, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:48.646584 12199 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260619 15:07:48.648211 12199 webserver.cc:460] Webserver started at http://127.24.103.67:35241/ using document root <none> and password file <none>
I20260619 15:07:48.648978 12199 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:48.649129 12199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:48.649451 12199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:48.652688 12199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "756faac5be1c4499a8de9364f2a078bf"
format_stamp: "Formatted at 2026-06-19 15:07:48 on dist-test-slave-zpbw"
server_key: "6da12e0081f84e2172aa890a93754094"
server_key_iv: "1dd74faa86dac11390866b8772daa338"
server_key_version: "encryptionkey@0"
I20260619 15:07:48.653586 12199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "756faac5be1c4499a8de9364f2a078bf"
format_stamp: "Formatted at 2026-06-19 15:07:48 on dist-test-slave-zpbw"
server_key: "6da12e0081f84e2172aa890a93754094"
server_key_iv: "1dd74faa86dac11390866b8772daa338"
server_key_version: "encryptionkey@0"
I20260619 15:07:48.659364 12199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 15:07:48.663143 12214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:48.664641 12199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:07:48.664863 12199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "756faac5be1c4499a8de9364f2a078bf"
format_stamp: "Formatted at 2026-06-19 15:07:48 on dist-test-slave-zpbw"
server_key: "6da12e0081f84e2172aa890a93754094"
server_key_iv: "1dd74faa86dac11390866b8772daa338"
server_key_version: "encryptionkey@0"
I20260619 15:07:48.665078 12199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:48.690907 12059 heartbeater.cc:503] Master 127.24.103.126:38169 was elected leader, sending a full tablet report...
I20260619 15:07:48.700702 12199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:48.707635 12199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:48.708055 12199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:48.709280 12199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:48.711387 12199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:48.711503 12199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:48.711649 12199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:48.711719 12199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:48.749902 12326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:37601 every 8 connection(s)
I20260619 15:07:48.749903 12199 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:37601
I20260619 15:07:48.752099 12199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:07:48.755956 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 12199
I20260619 15:07:48.756158 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:07:48.756880 24989 external_mini_cluster.cc:1428] Setting key 478b042aabd2640b5880a320b95f6abe
Jun 19 15:07:48 dist-test-slave-zpbw krb5kdc[11825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881668, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:07:48.779734 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.755251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:45261 (local address 127.24.103.126:38169)
0619 15:07:48.761268 (+ 6017us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:48.761273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:48.761289 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:07:48.761339 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:48.761342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:48.761436 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:48.761556 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:48.764042 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.764882 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.767841 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.768251 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.768423 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.772388 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:48.772393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:48.772397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:48.772433 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:48.775832 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:48.777004 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:48.777008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:48.777010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:48.777137 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:48.777701 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:48.777704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:48.777706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:48.777982 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:48.778101 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:48.779253 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0619 15:07:48.779447 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5894,"thread_start_us":5771,"threads_started":1}
I20260619 15:07:48.781558 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.755987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38169 (local address 127.24.103.67:45261)
0619 15:07:48.758492 (+ 2505us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:48.758552 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:48.760643 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:48.761924 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:48.761941 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:48.762887 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:48.763830 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.763860 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.765073 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.765082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.767647 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.767667 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.768559 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.768568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.768879 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.769736 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:07:48.769794 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:48.772110 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:48.776061 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:48.776074 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:48.776100 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:48.776864 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:48.777250 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:07:48.777257 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:07:48.777263 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:07:48.777574 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:07:48.778126 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:48.778138 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:48.778733 (+ 595us) client_negotiation.cc:769] Sending connection context
0619 15:07:48.779160 (+ 427us) client_negotiation.cc:240] Negotiation successful
0619 15:07:48.779870 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2178,"thread_start_us":1954,"threads_started":1}
I20260619 15:07:48.783236 12327 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38169
I20260619 15:07:48.783576 12327 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:48.784407 12327 heartbeater.cc:511] Master 127.24.103.126:38169 requested a full tablet report, sending...
I20260619 15:07:48.785946 11870 ts_manager.cc:194] Registered new tserver with Master: 756faac5be1c4499a8de9364f2a078bf (127.24.103.67:37601)
I20260619 15:07:48.786867 11870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:45261
I20260619 15:07:48.789445 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:07:48.813659 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.795724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.24.103.126:38169)
0619 15:07:48.795986 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:48.795995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:48.796402 (+ 407us) server_negotiation.cc:407] Connection header received
0619 15:07:48.796550 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:48.796558 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:48.796687 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:48.796843 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:48.798317 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.799770 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.801870 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.802725 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.802952 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.805854 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:48.805863 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:48.805871 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:48.805927 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:48.810153 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:48.811006 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:48.811015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:48.811021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:48.811213 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:48.811640 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:48.811648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:48.811654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:48.812226 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:48.812518 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:48.812898 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 15:07:48.813186 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 15:07:48.818982 11870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:07:48.833596 11870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:07:48.879556 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.849549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60134 (local address 127.24.103.66:42349)
0619 15:07:48.863327 (+ 13778us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:48.863335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:48.868321 (+ 4986us) server_negotiation.cc:407] Connection header received
0619 15:07:48.868620 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:48.868633 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:48.868945 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:48.869146 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:48.870165 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.871763 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.874799 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.876716 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.876857 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.877129 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:48.878901 (+ 1772us) server_negotiation.cc:299] Negotiation successful
0619 15:07:48.879138 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13584,"thread_start_us":13432,"threads_started":1}
I20260619 15:07:48.878742 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.849496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:42349 (local address 127.0.0.1:60134)
0619 15:07:48.868099 (+ 18603us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:48.868153 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:48.868374 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:48.869475 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:48.869482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:48.869511 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:48.869955 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.869974 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.871948 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.871956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.874563 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.874579 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.877393 (+ 2814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.877402 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.877737 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.877807 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 15:07:48.877960 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 15:07:48.878238 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18275,"mutex_wait_us":28,"thread_start_us":19166,"threads_started":1}
I20260619 15:07:48.880733 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.850357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.24.103.67:37601)
0619 15:07:48.867431 (+ 17074us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:48.867444 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:48.869151 (+ 1707us) server_negotiation.cc:407] Connection header received
0619 15:07:48.869454 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:48.869461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:48.869712 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:48.869899 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:48.872412 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.874067 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.877433 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.879056 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.879160 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.879387 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:48.880118 (+ 731us) server_negotiation.cc:299] Negotiation successful
0619 15:07:48.880325 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16843,"thread_start_us":16651,"threads_started":1}
I20260619 15:07:48.881727 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.849681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37601 (local address 127.0.0.1:55036)
0619 15:07:48.868987 (+ 19306us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:48.869007 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:48.869263 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:48.870328 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:48.870332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:48.870356 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:48.870800 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.870814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.874260 (+ 3446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.874265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.877225 (+ 2960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.877242 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.879594 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.879603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.879849 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.879903 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 15:07:48.881020 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0619 15:07:48.881209 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19188,"mutex_wait_us":583,"thread_start_us":17950,"threads_started":1}
I20260619 15:07:48.884794 12128 tablet_service.cc:1467] Processing CreateTablet for tablet f6f55b499c37491f882c9e8f69b8148c (DEFAULT_TABLE table=test-table [id=13fe9fb76c524936b9dc5d93984a53c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:48.885780 12262 tablet_service.cc:1467] Processing CreateTablet for tablet f6f55b499c37491f882c9e8f69b8148c (DEFAULT_TABLE table=test-table [id=13fe9fb76c524936b9dc5d93984a53c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:48.886549 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f55b499c37491f882c9e8f69b8148c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:48.887271 12262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f55b499c37491f882c9e8f69b8148c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:48.896514 12345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.869138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:39813 (local address 127.0.0.1:44500)
0619 15:07:48.873018 (+ 3880us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:48.873044 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:48.873199 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:48.874572 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:48.874580 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:48.874611 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:48.875273 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.875292 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.877420 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.877425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.892868 (+ 15443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.892895 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.895541 (+ 2646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.895549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.895732 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.895778 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 15:07:48.895870 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 15:07:48.896026 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3696,"mutex_wait_us":112,"thread_start_us":4186,"threads_started":1}
I20260619 15:07:48.897591 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.869148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44500 (local address 127.24.103.65:39813)
0619 15:07:48.874038 (+ 4890us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:48.874045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:48.874070 (+ 25us) server_negotiation.cc:407] Connection header received
0619 15:07:48.874153 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:48.874158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:48.874364 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:48.874643 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:48.875443 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.877250 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.893181 (+ 15931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.894938 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.895097 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.895344 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:48.897035 (+ 1691us) server_negotiation.cc:299] Negotiation successful
0619 15:07:48.897238 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4712,"thread_start_us":4545,"threads_started":1}
I20260619 15:07:48.902452 12346 tablet_bootstrap.cc:492] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf: Bootstrap starting.
I20260619 15:07:48.902099 11994 tablet_service.cc:1467] Processing CreateTablet for tablet f6f55b499c37491f882c9e8f69b8148c (DEFAULT_TABLE table=test-table [id=13fe9fb76c524936b9dc5d93984a53c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:48.903790 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f55b499c37491f882c9e8f69b8148c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:48.905844 12346 tablet_bootstrap.cc:654] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:48.906940 12346 log.cc:824] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:48.919144 12346 tablet_bootstrap.cc:492] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf: No bootstrap required, opened a new log
I20260619 15:07:48.919548 12346 ts_tablet_manager.cc:1397] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260619 15:07:48.925267 12348 tablet_bootstrap.cc:492] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05: Bootstrap starting.
I20260619 15:07:48.929008 12349 tablet_bootstrap.cc:492] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0: Bootstrap starting.
I20260619 15:07:48.929287 12348 tablet_bootstrap.cc:654] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:48.930717 12348 log.cc:824] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:48.934329 12346 raft_consensus.cc:348] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1361884e9b5b48a29c56fb77e3dc2e05" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42349 } } peers { permanent_uuid: "756faac5be1c4499a8de9364f2a078bf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37601 } } peers { permanent_uuid: "3f06eff8dee84455945bde563878d9d0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 39813 } }
I20260619 15:07:48.935214 12346 raft_consensus.cc:374] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:48.935407 12346 raft_consensus.cc:729] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 756faac5be1c4499a8de9364f2a078bf, State: Initialized, Role: FOLLOWER
I20260619 15:07:48.935696 12349 tablet_bootstrap.cc:654] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:48.936467 12346 consensus_queue.cc:260] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1361884e9b5b48a29c56fb77e3dc2e05" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42349 } } peers { permanent_uuid: "756faac5be1c4499a8de9364f2a078bf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37601 } } peers { permanent_uuid: "3f06eff8dee84455945bde563878d9d0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 39813 } }
I20260619 15:07:48.937326 12349 log.cc:824] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:48.940228 12346 ts_tablet_manager.cc:1428] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf: Time spent starting tablet: real 0.021s user 0.012s sys 0.005s
I20260619 15:07:48.940802 12327 heartbeater.cc:503] Master 127.24.103.126:38169 was elected leader, sending a full tablet report...
I20260619 15:07:48.941190 12348 tablet_bootstrap.cc:492] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05: No bootstrap required, opened a new log
I20260619 15:07:48.941512 12348 ts_tablet_manager.cc:1397] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20260619 15:07:48.944916 12349 tablet_bootstrap.cc:492] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0: No bootstrap required, opened a new log
I20260619 15:07:48.946393 12349 ts_tablet_manager.cc:1397] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20260619 15:07:48.952926 12348 raft_consensus.cc:348] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1361884e9b5b48a29c56fb77e3dc2e05" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42349 } } peers { permanent_uuid: "756faac5be1c4499a8de9364f2a078bf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37601 } } peers { permanent_uuid: "3f06eff8dee84455945bde563878d9d0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 39813 } }
I20260619 15:07:48.953382 12348 raft_consensus.cc:374] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:48.953510 12348 raft_consensus.cc:729] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1361884e9b5b48a29c56fb77e3dc2e05, State: Initialized, Role: FOLLOWER
I20260619 15:07:48.954157 12348 consensus_queue.cc:260] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1361884e9b5b48a29c56fb77e3dc2e05" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42349 } } peers { permanent_uuid: "756faac5be1c4499a8de9364f2a078bf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37601 } } peers { permanent_uuid: "3f06eff8dee84455945bde563878d9d0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 39813 } }
I20260619 15:07:48.955420 12193 heartbeater.cc:503] Master 127.24.103.126:38169 was elected leader, sending a full tablet report...
I20260619 15:07:48.956229 12348 ts_tablet_manager.cc:1428] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05: Time spent starting tablet: real 0.015s user 0.001s sys 0.011s
I20260619 15:07:48.959163 12349 raft_consensus.cc:348] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1361884e9b5b48a29c56fb77e3dc2e05" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42349 } } peers { permanent_uuid: "756faac5be1c4499a8de9364f2a078bf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37601 } } peers { permanent_uuid: "3f06eff8dee84455945bde563878d9d0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 39813 } }
I20260619 15:07:48.959686 12349 raft_consensus.cc:374] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:48.959793 12349 raft_consensus.cc:729] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f06eff8dee84455945bde563878d9d0, State: Initialized, Role: FOLLOWER
I20260619 15:07:48.960438 12349 consensus_queue.cc:260] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1361884e9b5b48a29c56fb77e3dc2e05" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42349 } } peers { permanent_uuid: "756faac5be1c4499a8de9364f2a078bf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37601 } } peers { permanent_uuid: "3f06eff8dee84455945bde563878d9d0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 39813 } }
I20260619 15:07:48.963541 12349 ts_tablet_manager.cc:1428] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260619 15:07:48.981791 12351 raft_consensus.cc:482] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:07:48.982129 12351 raft_consensus.cc:504] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1361884e9b5b48a29c56fb77e3dc2e05" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42349 } } peers { permanent_uuid: "756faac5be1c4499a8de9364f2a078bf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37601 } } peers { permanent_uuid: "3f06eff8dee84455945bde563878d9d0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 39813 } }
I20260619 15:07:48.983974 12351 leader_election.cc:288] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1361884e9b5b48a29c56fb77e3dc2e05 (127.24.103.66:42349), 3f06eff8dee84455945bde563878d9d0 (127.24.103.65:39813)
I20260619 15:07:48.991200 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.984617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:52983 (local address 127.24.103.66:42349)
0619 15:07:48.984830 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:48.984837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:48.985204 (+ 367us) server_negotiation.cc:407] Connection header received
0619 15:07:48.985309 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:48.985317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:48.985409 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:48.985496 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:48.986212 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.987318 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.989285 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.990174 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.990344 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.990422 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:48.990768 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 15:07:48.990931 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:07:48.991255 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.984475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:42349 (local address 127.24.103.67:52983)
0619 15:07:48.984894 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:48.984918 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:48.985071 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:48.985790 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:48.985796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:48.985831 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:48.986046 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.986055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.987482 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.987488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.989114 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.989126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.990345 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.990351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.990591 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.990617 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 15:07:48.990735 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 15:07:48.990917 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":2}
I20260619 15:07:48.992663 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f55b499c37491f882c9e8f69b8148c" candidate_uuid: "756faac5be1c4499a8de9364f2a078bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1361884e9b5b48a29c56fb77e3dc2e05" is_pre_election: true
I20260619 15:07:48.993296 12148 raft_consensus.cc:2454] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 756faac5be1c4499a8de9364f2a078bf in term 0.
I20260619 15:07:48.994168 12217 leader_election.cc:302] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1361884e9b5b48a29c56fb77e3dc2e05, 756faac5be1c4499a8de9364f2a078bf; no voters:
I20260619 15:07:48.994553 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.985235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:52977 (local address 127.24.103.65:39813)
0619 15:07:48.985828 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:48.985837 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:48.985858 (+ 21us) server_negotiation.cc:407] Connection header received
0619 15:07:48.986060 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:48.986068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:48.986154 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:48.986291 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:48.987260 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.988343 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.991611 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.993319 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.993511 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.993615 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:48.994027 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 15:07:48.994207 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395}
I20260619 15:07:48.994956 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:48.984582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:39813 (local address 127.24.103.67:52977)
0619 15:07:48.985780 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:48.985796 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:48.985913 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:48.986318 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:48.986321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:48.986348 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:48.987069 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.987081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.988505 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.988513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.991351 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:48.991371 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:48.993610 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:48.993618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:48.993834 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:48.993873 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 15:07:48.993976 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 15:07:48.994125 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":2,"thread_start_us":916,"threads_started":1}
I20260619 15:07:48.994890 12351 raft_consensus.cc:2781] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:07:48.995081 12351 raft_consensus.cc:482] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:07:48.995204 12351 raft_consensus.cc:3037] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:48.996043 12014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f55b499c37491f882c9e8f69b8148c" candidate_uuid: "756faac5be1c4499a8de9364f2a078bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f06eff8dee84455945bde563878d9d0" is_pre_election: true
I20260619 15:07:48.996644 12014 raft_consensus.cc:2454] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 756faac5be1c4499a8de9364f2a078bf in term 0.
I20260619 15:07:48.998256 12351 raft_consensus.cc:504] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1361884e9b5b48a29c56fb77e3dc2e05" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42349 } } peers { permanent_uuid: "756faac5be1c4499a8de9364f2a078bf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37601 } } peers { permanent_uuid: "3f06eff8dee84455945bde563878d9d0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 39813 } }
I20260619 15:07:48.999009 12351 leader_election.cc:288] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [CANDIDATE]: Term 1 election: Requested vote from peers 1361884e9b5b48a29c56fb77e3dc2e05 (127.24.103.66:42349), 3f06eff8dee84455945bde563878d9d0 (127.24.103.65:39813)
I20260619 15:07:48.999512 12014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f55b499c37491f882c9e8f69b8148c" candidate_uuid: "756faac5be1c4499a8de9364f2a078bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f06eff8dee84455945bde563878d9d0"
I20260619 15:07:48.999581 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f55b499c37491f882c9e8f69b8148c" candidate_uuid: "756faac5be1c4499a8de9364f2a078bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1361884e9b5b48a29c56fb77e3dc2e05"
I20260619 15:07:48.999737 12014 raft_consensus.cc:3037] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:48.999852 12148 raft_consensus.cc:3037] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:49.002043 12014 raft_consensus.cc:2454] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 756faac5be1c4499a8de9364f2a078bf in term 1.
I20260619 15:07:49.002545 12215 leader_election.cc:302] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f06eff8dee84455945bde563878d9d0, 756faac5be1c4499a8de9364f2a078bf; no voters:
I20260619 15:07:49.002914 12351 raft_consensus.cc:2781] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:49.003479 12351 raft_consensus.cc:686] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 1 LEADER]: Becoming Leader. State: Replica: 756faac5be1c4499a8de9364f2a078bf, State: Running, Role: LEADER
I20260619 15:07:49.003808 12148 raft_consensus.cc:2454] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 756faac5be1c4499a8de9364f2a078bf in term 1.
W20260619 15:07:49.004042 12328 tablet.cc:2369] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:07:49.004073 12351 consensus_queue.cc:237] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1361884e9b5b48a29c56fb77e3dc2e05" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42349 } } peers { permanent_uuid: "756faac5be1c4499a8de9364f2a078bf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37601 } } peers { permanent_uuid: "3f06eff8dee84455945bde563878d9d0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 39813 } }
I20260619 15:07:49.008849 11869 catalog_manager.cc:5495] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 756faac5be1c4499a8de9364f2a078bf (127.24.103.67). New cstate: current_term: 1 leader_uuid: "756faac5be1c4499a8de9364f2a078bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1361884e9b5b48a29c56fb77e3dc2e05" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "756faac5be1c4499a8de9364f2a078bf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f06eff8dee84455945bde563878d9d0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 39813 } health_report { overall_health: UNKNOWN } } }
I20260619 15:07:49.034191 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:49.027175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.24.103.67:37601)
0619 15:07:49.027363 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:49.027370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:49.027472 (+ 102us) server_negotiation.cc:407] Connection header received
0619 15:07:49.027711 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:49.027715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:49.027810 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:49.027948 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:07:49.028455 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:49.029184 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:49.031650 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:49.032059 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:49.032155 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:49.032892 (+ 737us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:07:49.033600 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:07:49.033760 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:49.033884 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 15:07:49.033957 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 15:07:49.042711 12148 raft_consensus.cc:1261] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05 [term 1 FOLLOWER]: Refusing update from remote peer 756faac5be1c4499a8de9364f2a078bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:07:49.043064 12014 raft_consensus.cc:1261] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0 [term 1 FOLLOWER]: Refusing update from remote peer 756faac5be1c4499a8de9364f2a078bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:07:49.043905 12356 consensus_queue.cc:1035] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1361884e9b5b48a29c56fb77e3dc2e05" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:07:49.044348 12351 consensus_queue.cc:1035] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f06eff8dee84455945bde563878d9d0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:07:49.060467 12359 mvcc.cc:205] Tried to move back new op lower bound from 7298587316388327424 to 7298587316245172224. Current Snapshot: MvccSnapshot[applied={T|T < 7298587316388327424}]
I20260619 15:07:49.082289 11870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:07:49.082614 11870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:07:49.086726 11870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 966dc5a7d66846e8b764105b8ac10978: Sending DeleteTablet for 3 replicas of tablet f6f55b499c37491f882c9e8f69b8148c
I20260619 15:07:49.088160 12128 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f55b499c37491f882c9e8f69b8148c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:49 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:60134
I20260619 15:07:49.088171 12262 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f55b499c37491f882c9e8f69b8148c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:49 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:55036
I20260619 15:07:49.088604 11994 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f55b499c37491f882c9e8f69b8148c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:49 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:44500
I20260619 15:07:49.090273 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 11931
I20260619 15:07:49.100884 12370 tablet_replica.cc:330] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0: stopping tablet replica
I20260619 15:07:49.101545 12370 raft_consensus.cc:2229] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:07:49.102262 12370 raft_consensus.cc:2258] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:49.105839 12370 ts_tablet_manager.cc:1905] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:49.109196 12368 tablet_replica.cc:330] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05: stopping tablet replica
I20260619 15:07:49.109881 12368 raft_consensus.cc:2229] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:07:49.110594 12368 raft_consensus.cc:2258] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:49.114089 12370 ts_tablet_manager.cc:1918] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:49.114320 12370 log.cc:1196] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/f6f55b499c37491f882c9e8f69b8148c
I20260619 15:07:49.115001 12370 ts_tablet_manager.cc:1939] T f6f55b499c37491f882c9e8f69b8148c P 3f06eff8dee84455945bde563878d9d0: Deleting consensus metadata
I20260619 15:07:49.117156 11854 catalog_manager.cc:4841] TS 3f06eff8dee84455945bde563878d9d0 (127.24.103.65:39813): tablet f6f55b499c37491f882c9e8f69b8148c (table test-table [id=13fe9fb76c524936b9dc5d93984a53c7]) successfully deleted
I20260619 15:07:49.117615 12369 tablet_replica.cc:330] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf: stopping tablet replica
I20260619 15:07:49.118127 12368 ts_tablet_manager.cc:1905] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:49.118103 12369 raft_consensus.cc:2229] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:07:49.118940 12369 raft_consensus.cc:2258] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:49.121222 12369 ts_tablet_manager.cc:1905] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 15:07:49 dist-test-slave-zpbw krb5kdc[11825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260619 15:07:49.126291 12368 ts_tablet_manager.cc:1918] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:49.126493 12368 log.cc:1196] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/f6f55b499c37491f882c9e8f69b8148c
I20260619 15:07:49.127147 12368 ts_tablet_manager.cc:1939] T f6f55b499c37491f882c9e8f69b8148c P 1361884e9b5b48a29c56fb77e3dc2e05: Deleting consensus metadata
I20260619 15:07:49.129040 12369 ts_tablet_manager.cc:1918] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:49.129251 12369 log.cc:1196] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/f6f55b499c37491f882c9e8f69b8148c
I20260619 15:07:49.129307 11856 catalog_manager.cc:4841] TS 1361884e9b5b48a29c56fb77e3dc2e05 (127.24.103.66:42349): tablet f6f55b499c37491f882c9e8f69b8148c (table test-table [id=13fe9fb76c524936b9dc5d93984a53c7]) successfully deleted
I20260619 15:07:49.129843 12369 ts_tablet_manager.cc:1939] T f6f55b499c37491f882c9e8f69b8148c P 756faac5be1c4499a8de9364f2a078bf: Deleting consensus metadata
I20260619 15:07:49.131748 11854 catalog_manager.cc:4841] TS 756faac5be1c4499a8de9364f2a078bf (127.24.103.67:37601): tablet f6f55b499c37491f882c9e8f69b8148c (table test-table [id=13fe9fb76c524936b9dc5d93984a53c7]) successfully deleted
I20260619 15:07:49.132709 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:49.091677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44508 (local address 127.24.103.65:39813)
0619 15:07:49.091867 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:49.091874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:49.103838 (+ 11964us) server_negotiation.cc:407] Connection header received
0619 15:07:49.105165 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:49.105176 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:49.105310 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:49.105411 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:49.111312 (+ 5901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:49.112923 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:49.117710 (+ 4787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:49.118390 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:49.118504 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:49.124656 (+ 6152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:49.124677 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:49.124698 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:49.124754 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:49.129018 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:49.130158 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:49.130168 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:49.130171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:49.130330 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:49.130834 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:49.130839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:49.130843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:49.131219 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:49.131558 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:49.132116 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 15:07:49.132377 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 15:07:49.133513 12054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:49.236599 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 11931
I20260619 15:07:49.255622 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 12065
Jun 19 15:07:49 dist-test-slave-zpbw krb5kdc[11825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260619 15:07:49.273417 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:49.256484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.24.103.66:42349)
0619 15:07:49.256693 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:49.256700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:49.256722 (+ 22us) server_negotiation.cc:407] Connection header received
0619 15:07:49.256870 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:49.256876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:49.256967 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:49.257076 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:49.258411 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:49.259634 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:49.261117 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:49.261750 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:49.261883 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:49.266556 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:49.266576 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:49.266595 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:49.266659 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:49.270431 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:49.271251 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:49.271260 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:49.271264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:49.271423 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:49.271780 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:49.271785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:49.271789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:49.272271 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:49.272568 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:49.272783 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 15:07:49.273023 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 15:07:49.274183 12188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:49.364918 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 12065
I20260619 15:07:49.383127 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 12199
Jun 19 15:07:49 dist-test-slave-zpbw krb5kdc[11825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260619 15:07:49.401189 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:49.384242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.24.103.67:37601)
0619 15:07:49.384453 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:49.384458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:49.384474 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:07:49.384527 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:49.384530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:49.384611 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:49.384727 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:49.386180 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:49.387059 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:49.388419 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:49.388869 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:49.389030 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:49.394332 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:49.394347 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:49.394363 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:49.394408 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:49.397901 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:49.398821 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:49.398834 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:49.398837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:49.399013 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:49.399436 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:49.399441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:49.399444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:49.399883 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:49.400210 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:49.400549 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 15:07:49.400788 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 15:07:49.402143 12322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:49.525978 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 12199
I20260619 15:07:49.545289 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 11838
I20260619 15:07:49.546418 11900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:49.659698 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 11838
2026-06-19T15:07:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4479 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T15:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:07:49Z Disabled control of system clock
I20260619 15:07:49.739620 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38067
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:07:50.096597 12379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:50.096948 12379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:50.097036 12379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:50.104710 12379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:07:50.104846 12379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:50.104889 12379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:50.104946 12379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:07:50.105000 12379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:07:50.114926 12379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:50.116318 12379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:50.118247 12379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:50.131333 12384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:50.131464 12387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:50.131487 12385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:50.132628 12379 server_base.cc:1034] running on GCE node
I20260619 15:07:50.133526 12379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:50.135327 12379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:50.136540 12379 hybrid_clock.cc:648] HybridClock initialized: now 1781881670136491 us; error 43 us; skew 500 ppm
I20260619 15:07:50.136955 12379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:07:50.139508 12379 webserver.cc:460] Webserver started at http://127.24.103.126:34007/ using document root <none> and password file <none>
I20260619 15:07:50.140290 12379 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:50.140419 12379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:50.140781 12379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:50.143996 12379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "28dbe960ad78467e86102ebcb700f965"
format_stamp: "Formatted at 2026-06-19 15:07:50 on dist-test-slave-zpbw"
server_key: "f5a21514d19696ad91373f789ca06b91"
server_key_iv: "085cdf9ac5708e4cfad32c6492bd95cd"
server_key_version: "encryptionkey@0"
I20260619 15:07:50.144893 12379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "28dbe960ad78467e86102ebcb700f965"
format_stamp: "Formatted at 2026-06-19 15:07:50 on dist-test-slave-zpbw"
server_key: "f5a21514d19696ad91373f789ca06b91"
server_key_iv: "085cdf9ac5708e4cfad32c6492bd95cd"
server_key_version: "encryptionkey@0"
I20260619 15:07:50.150535 12379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:07:50.154345 12393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:50.155985 12379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:07:50.156208 12379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "28dbe960ad78467e86102ebcb700f965"
format_stamp: "Formatted at 2026-06-19 15:07:50 on dist-test-slave-zpbw"
server_key: "f5a21514d19696ad91373f789ca06b91"
server_key_iv: "085cdf9ac5708e4cfad32c6492bd95cd"
server_key_version: "encryptionkey@0"
I20260619 15:07:50.156452 12379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:50.175240 12379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:50.176165 12379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:50.176476 12379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:50.195894 12379 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:38067
I20260619 15:07:50.195914 12444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:38067 every 8 connection(s)
I20260619 15:07:50.197394 12379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:07:50.201372 12445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:50.206633 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 12379
I20260619 15:07:50.206864 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:07:50.207326 24989 external_mini_cluster.cc:1428] Setting key df883f3efbbcbc87bb1d1552b68a41bb
I20260619 15:07:50.213011 12445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965: Bootstrap starting.
I20260619 15:07:50.216235 12445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:50.217344 12445 log.cc:824] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:50.220832 12445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965: No bootstrap required, opened a new log
I20260619 15:07:50.221681 12448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:50.209234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.24.103.126:38067)
0619 15:07:50.210338 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:50.210360 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:50.210422 (+ 62us) server_negotiation.cc:407] Connection header received
0619 15:07:50.212331 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:50.212355 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:50.212863 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:50.213266 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:50.213860 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:50.215242 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:50.216875 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:50.217663 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:50.217889 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:50.219232 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:50.219240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:50.219254 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:50.219823 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:50.219906 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:50.219943 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:50.220033 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:50.220304 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 15:07:50.220650 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":408,"threads_started":1}
I20260619 15:07:50.229884 12445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28dbe960ad78467e86102ebcb700f965" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38067 } }
I20260619 15:07:50.230281 12445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:50.230418 12445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dbe960ad78467e86102ebcb700f965, State: Initialized, Role: FOLLOWER
I20260619 15:07:50.231081 12445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28dbe960ad78467e86102ebcb700f965" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38067 } }
I20260619 15:07:50.231350 12445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:07:50.231506 12445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:07:50.231703 12445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:50.234009 12445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28dbe960ad78467e86102ebcb700f965" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38067 } }
I20260619 15:07:50.234561 12445 leader_election.cc:302] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28dbe960ad78467e86102ebcb700f965; no voters:
I20260619 15:07:50.235330 12445 leader_election.cc:288] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:07:50.235481 12450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:50.236691 12450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [term 1 LEADER]: Becoming Leader. State: Replica: 28dbe960ad78467e86102ebcb700f965, State: Running, Role: LEADER
I20260619 15:07:50.237221 12450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28dbe960ad78467e86102ebcb700f965" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38067 } }
I20260619 15:07:50.237669 12445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:07:50.240089 12452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28dbe960ad78467e86102ebcb700f965. Latest consensus state: current_term: 1 leader_uuid: "28dbe960ad78467e86102ebcb700f965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28dbe960ad78467e86102ebcb700f965" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38067 } } }
I20260619 15:07:50.240581 12452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:50.240219 12451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28dbe960ad78467e86102ebcb700f965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28dbe960ad78467e86102ebcb700f965" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38067 } } }
I20260619 15:07:50.241091 12451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:50.244040 12457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:07:50.248483 12457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:07:50.258134 12457 catalog_manager.cc:1347] Generated new cluster ID: 3df2d5134042482282da61b0dc21d5f3
I20260619 15:07:50.258292 12457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:07:50.274168 12457 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:07:50.275624 12457 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:07:50.297257 12457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965: Generated new TSK 0
I20260619 15:07:50.298242 12457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:07:50.312359 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38067
--builtin_ntp_servers=127.24.103.84:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:07:50.659819 12469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:50.660198 12469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:50.660363 12469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:50.668200 12469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:50.668324 12469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:50.668503 12469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260619 15:07:50.678922 12469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:50.680469 12469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:50.682353 12469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:50.695814 12475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:50.695856 12477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:50.696161 12474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:50.697059 12469 server_base.cc:1034] running on GCE node
I20260619 15:07:50.697798 12469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:50.698959 12469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:50.700197 12469 hybrid_clock.cc:648] HybridClock initialized: now 1781881670700128 us; error 68 us; skew 500 ppm
I20260619 15:07:50.700549 12469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:07:50.707186 12469 webserver.cc:460] Webserver started at http://127.24.103.65:39277/ using document root <none> and password file <none>
I20260619 15:07:50.708022 12469 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:50.708171 12469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:50.708510 12469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:50.711771 12469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/data/instance:
uuid: "c5f18e0e5eaa40bd8f2b84333da902ad"
format_stamp: "Formatted at 2026-06-19 15:07:50 on dist-test-slave-zpbw"
server_key: "d5f6617f805b89f52751259266d89617"
server_key_iv: "d7edb2256faa557232bcc4789e79ead2"
server_key_version: "encryptionkey@0"
I20260619 15:07:50.712679 12469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "c5f18e0e5eaa40bd8f2b84333da902ad"
format_stamp: "Formatted at 2026-06-19 15:07:50 on dist-test-slave-zpbw"
server_key: "d5f6617f805b89f52751259266d89617"
server_key_iv: "d7edb2256faa557232bcc4789e79ead2"
server_key_version: "encryptionkey@0"
I20260619 15:07:50.718446 12469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 15:07:50.722462 12483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:50.723973 12469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:07:50.724174 12469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/wal
uuid: "c5f18e0e5eaa40bd8f2b84333da902ad"
format_stamp: "Formatted at 2026-06-19 15:07:50 on dist-test-slave-zpbw"
server_key: "d5f6617f805b89f52751259266d89617"
server_key_iv: "d7edb2256faa557232bcc4789e79ead2"
server_key_version: "encryptionkey@0"
I20260619 15:07:50.724413 12469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:50.747442 12469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:50.748378 12469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:50.748720 12469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:50.749739 12469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:50.751437 12469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:50.751567 12469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:50.751710 12469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:50.751809 12469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:50.786617 12469 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:33141
I20260619 15:07:50.786708 12595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:33141 every 8 connection(s)
I20260619 15:07:50.788038 12469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/data/info.pb
I20260619 15:07:50.791674 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 12469
I20260619 15:07:50.791895 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/wal/instance
I20260619 15:07:50.792451 24989 external_mini_cluster.cc:1428] Setting key ffdc4b55aa71a3df0d7b0fb84cf2bc3d
I20260619 15:07:50.795321 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38067
--builtin_ntp_servers=127.24.103.84:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 15:07:50.802608 12598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:50.790889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:44023 (local address 127.24.103.126:38067)
0619 15:07:50.791366 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:50.791371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:50.792661 (+ 1290us) server_negotiation.cc:407] Connection header received
0619 15:07:50.794739 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:50.794744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:50.794866 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:50.794990 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:50.796335 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:50.797574 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:50.799270 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:50.800083 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:50.800234 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:50.801708 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:50.801713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:50.801717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:50.801737 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:50.801785 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:50.801798 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:50.801897 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:50.802143 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 15:07:50.802248 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":252,"threads_started":1}
I20260619 15:07:50.804070 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:50.791392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38067 (local address 127.24.103.65:44023)
0619 15:07:50.792464 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:50.792502 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:50.794467 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:50.795245 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:50.795255 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:50.795362 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:50.796102 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:50.796132 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:50.797735 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:50.797740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:50.799107 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:50.799120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:50.800311 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:50.800316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:50.800558 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:50.801336 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:07:50.801339 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:50.801383 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:07:50.801494 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:50.801898 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:50.801911 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:50.801922 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 15:07:50.802110 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 15:07:50.802551 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":359,"threads_started":1}
I20260619 15:07:50.805809 12596 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38067
I20260619 15:07:50.806162 12596 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:50.807032 12596 heartbeater.cc:511] Master 127.24.103.126:38067 requested a full tablet report, sending...
I20260619 15:07:50.809636 12410 ts_manager.cc:194] Registered new tserver with Master: c5f18e0e5eaa40bd8f2b84333da902ad (127.24.103.65:33141)
I20260619 15:07:50.811565 12410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.65:44023
W20260619 15:07:51.139848 12600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:51.140205 12600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:51.140376 12600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:51.147828 12600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:51.148010 12600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:51.148200 12600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260619 15:07:51.157855 12600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:51.159686 12600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:51.161943 12600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:51.179502 12606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:51.179591 12608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:51.179495 12605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:51.181008 12600 server_base.cc:1034] running on GCE node
I20260619 15:07:51.181828 12600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:51.183290 12600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:51.184552 12600 hybrid_clock.cc:648] HybridClock initialized: now 1781881671184473 us; error 51 us; skew 500 ppm
I20260619 15:07:51.184927 12600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:07:51.188158 12600 webserver.cc:460] Webserver started at http://127.24.103.66:34253/ using document root <none> and password file <none>
I20260619 15:07:51.189213 12600 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:51.189368 12600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:51.189752 12600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:51.194182 12600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/data/instance:
uuid: "1aad286385914eebbd9cdc1d2bfb6238"
format_stamp: "Formatted at 2026-06-19 15:07:51 on dist-test-slave-zpbw"
server_key: "0a3e3aeb05c98e9ecfcdbf2b448cb946"
server_key_iv: "1f5ee2f196c3d37a8bc138dd212654a3"
server_key_version: "encryptionkey@0"
I20260619 15:07:51.195528 12600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "1aad286385914eebbd9cdc1d2bfb6238"
format_stamp: "Formatted at 2026-06-19 15:07:51 on dist-test-slave-zpbw"
server_key: "0a3e3aeb05c98e9ecfcdbf2b448cb946"
server_key_iv: "1f5ee2f196c3d37a8bc138dd212654a3"
server_key_version: "encryptionkey@0"
I20260619 15:07:51.203680 12600 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260619 15:07:51.209497 12614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:51.211500 12600 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 15:07:51.211782 12600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/wal
uuid: "1aad286385914eebbd9cdc1d2bfb6238"
format_stamp: "Formatted at 2026-06-19 15:07:51 on dist-test-slave-zpbw"
server_key: "0a3e3aeb05c98e9ecfcdbf2b448cb946"
server_key_iv: "1f5ee2f196c3d37a8bc138dd212654a3"
server_key_version: "encryptionkey@0"
I20260619 15:07:51.212185 12600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:51.231969 12600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:51.232870 12600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:51.233194 12600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:51.234305 12600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:51.235924 12600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:51.236018 12600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:51.236166 12600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:51.236255 12600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:51.270686 12600 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:43891
I20260619 15:07:51.270738 12726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:43891 every 8 connection(s)
I20260619 15:07:51.272089 12600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/data/info.pb
I20260619 15:07:51.275242 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 12600
I20260619 15:07:51.275449 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/wal/instance
I20260619 15:07:51.276043 24989 external_mini_cluster.cc:1428] Setting key 201410c12fe3a4b4e5e795016ea6936c
I20260619 15:07:51.279138 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38067
--builtin_ntp_servers=127.24.103.84:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 15:07:51.285673 12598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.274734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:46817 (local address 127.24.103.126:38067)
0619 15:07:51.274964 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:51.274969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:51.276720 (+ 1751us) server_negotiation.cc:407] Connection header received
0619 15:07:51.278902 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:51.278908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:51.279006 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:51.279121 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:51.280313 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.281032 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.282678 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.283176 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.283340 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.284827 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:51.284833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:51.284845 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:51.284865 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:51.284898 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:51.284911 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:51.285008 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:51.285244 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 15:07:51.285364 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 15:07:51.286942 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.275471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38067 (local address 127.24.103.66:46817)
0619 15:07:51.276529 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:51.276574 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:51.278527 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:51.279392 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:51.279402 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:51.279506 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:51.280125 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.280143 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.281197 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.281209 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.282520 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.282533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.283454 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.283462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.283704 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.284463 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:07:51.284466 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:51.284510 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:07:51.284610 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:51.285008 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:51.285022 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:51.285032 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 15:07:51.285236 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 15:07:51.285682 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":322,"threads_started":1}
I20260619 15:07:51.288434 12727 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38067
I20260619 15:07:51.288774 12727 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:51.289783 12727 heartbeater.cc:511] Master 127.24.103.126:38067 requested a full tablet report, sending...
I20260619 15:07:51.291565 12410 ts_manager.cc:194] Registered new tserver with Master: 1aad286385914eebbd9cdc1d2bfb6238 (127.24.103.66:43891)
I20260619 15:07:51.292405 12410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.66:46817
W20260619 15:07:51.624173 12730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:51.624540 12730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:51.624686 12730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:51.632514 12730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:51.632642 12730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:51.632838 12730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260619 15:07:51.642736 12730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:51.644186 12730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:51.646130 12730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:51.660645 12735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:51.660687 12736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:51.663235 12738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:51.665238 12730 server_base.cc:1034] running on GCE node
I20260619 15:07:51.665977 12730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:51.667143 12730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:51.668356 12730 hybrid_clock.cc:648] HybridClock initialized: now 1781881671668312 us; error 38 us; skew 500 ppm
I20260619 15:07:51.668713 12730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 15:07:51.671451 12730 webserver.cc:460] Webserver started at http://127.24.103.67:42267/ using document root <none> and password file <none>
I20260619 15:07:51.672230 12730 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:51.672333 12730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:51.672703 12730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:51.675889 12730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/data/instance:
uuid: "469f646c24cb4743b04b5aa84592a534"
format_stamp: "Formatted at 2026-06-19 15:07:51 on dist-test-slave-zpbw"
server_key: "553dd20b974b1572c7700e44457eeddc"
server_key_iv: "04599a8637f2146517214df8b1615ced"
server_key_version: "encryptionkey@0"
I20260619 15:07:51.676764 12730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "469f646c24cb4743b04b5aa84592a534"
format_stamp: "Formatted at 2026-06-19 15:07:51 on dist-test-slave-zpbw"
server_key: "553dd20b974b1572c7700e44457eeddc"
server_key_iv: "04599a8637f2146517214df8b1615ced"
server_key_version: "encryptionkey@0"
I20260619 15:07:51.682498 12730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:07:51.686571 12744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:51.688117 12730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:07:51.688325 12730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/wal
uuid: "469f646c24cb4743b04b5aa84592a534"
format_stamp: "Formatted at 2026-06-19 15:07:51 on dist-test-slave-zpbw"
server_key: "553dd20b974b1572c7700e44457eeddc"
server_key_iv: "04599a8637f2146517214df8b1615ced"
server_key_version: "encryptionkey@0"
I20260619 15:07:51.688576 12730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:51.726056 12730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:51.727206 12730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:51.727603 12730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:51.728781 12730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:07:51.730319 12730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:07:51.730441 12730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:51.730585 12730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:07:51.730675 12730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:51.764880 12730 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:38623
I20260619 15:07:51.764946 12856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:38623 every 8 connection(s)
I20260619 15:07:51.766302 12730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/data/info.pb
I20260619 15:07:51.768031 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 12730
I20260619 15:07:51.768210 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/wal/instance
I20260619 15:07:51.768769 24989 external_mini_cluster.cc:1428] Setting key 7f17f821bd613f58ed5a246e6f54c7f6
I20260619 15:07:51.780431 12598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.769671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:35343 (local address 127.24.103.126:38067)
0619 15:07:51.769910 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:51.769916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:51.771600 (+ 1684us) server_negotiation.cc:407] Connection header received
0619 15:07:51.773756 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:51.773763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:51.773862 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:51.773972 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:51.775151 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.775895 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.777577 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.777987 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.778121 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.779530 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:51.779540 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:51.779543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:51.779562 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:51.779593 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:51.779605 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:51.779699 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:51.779928 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 15:07:51.780033 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 15:07:51.781476 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.770425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38067 (local address 127.24.103.67:35343)
0619 15:07:51.771460 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:51.771496 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:51.773468 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:51.774209 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:51.774218 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:51.774321 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:07:51.774990 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.775008 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.776055 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.776065 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.777432 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.777445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.778223 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.778231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.778471 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.779180 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:07:51.779182 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:07:51.779223 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:07:51.779329 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:07:51.779697 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:07:51.779709 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:07:51.779719 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 15:07:51.779903 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 15:07:51.780337 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":345,"threads_started":1}
I20260619 15:07:51.783041 12857 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38067
I20260619 15:07:51.783411 12857 heartbeater.cc:464] Registering TS with master...
I20260619 15:07:51.784334 12857 heartbeater.cc:511] Master 127.24.103.126:38067 requested a full tablet report, sending...
I20260619 15:07:51.785707 12410 ts_manager.cc:194] Registered new tserver with Master: 469f646c24cb4743b04b5aa84592a534 (127.24.103.67:38623)
I20260619 15:07:51.786865 12410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.67:35343
I20260619 15:07:51.797240 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:07:51.807379 12598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.801522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.24.103.126:38067)
0619 15:07:51.801710 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:51.801715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:51.802269 (+ 554us) server_negotiation.cc:407] Connection header received
0619 15:07:51.802409 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:51.802413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:51.802493 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:51.802589 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:51.803136 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.803901 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.805246 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.805662 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.805763 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.806719 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:51.806723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:51.806726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:51.806746 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:51.806844 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:51.806858 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:51.806930 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:51.807056 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 15:07:51.807132 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:07:51.814888 12596 heartbeater.cc:503] Master 127.24.103.126:38067 was elected leader, sending a full tablet report...
I20260619 15:07:51.811843 12410 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:07:51.820811 12410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:07:51.849949 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.835107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.24.103.67:38623)
0619 15:07:51.840227 (+ 5120us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:51.840240 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:51.840741 (+ 501us) server_negotiation.cc:407] Connection header received
0619 15:07:51.840881 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:51.840890 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:51.841188 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:51.841478 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:51.843211 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.844823 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.847194 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.848669 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.848842 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.849158 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:51.849394 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 15:07:51.849573 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4880,"thread_start_us":4652,"threads_started":1}
I20260619 15:07:51.849987 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.834543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:38623 (local address 127.0.0.1:47960)
0619 15:07:51.840470 (+ 5927us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:51.840493 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:51.840641 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:51.841691 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:51.841695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:51.841714 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:51.842917 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.842933 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.844948 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.844955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.847001 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.847018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.848895 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.848901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.849104 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.849167 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 15:07:51.849336 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 15:07:51.849546 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5768,"mutex_wait_us":4,"thread_start_us":929,"threads_started":1}
I20260619 15:07:51.850518 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.834346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:33141 (local address 127.0.0.1:60142)
0619 15:07:51.835511 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:51.835564 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:51.835818 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:51.841442 (+ 5624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:51.841450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:51.841487 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:51.841995 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.842016 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.844619 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.844627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.847266 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.847281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.849613 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.849619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.849796 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.849836 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 15:07:51.849940 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 15:07:51.850094 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":75,"thread_start_us":5813,"threads_started":1}
I20260619 15:07:51.851450 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.834895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.24.103.65:33141)
0619 15:07:51.840776 (+ 5881us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:51.840792 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:51.840816 (+ 24us) server_negotiation.cc:407] Connection header received
0619 15:07:51.840900 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:51.840905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:51.841163 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:51.841404 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:51.842304 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.844489 (+ 2185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.847494 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.849057 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.849186 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.849473 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:51.850896 (+ 1423us) server_negotiation.cc:299] Negotiation successful
0619 15:07:51.851076 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5686,"thread_start_us":5540,"threads_started":1}
I20260619 15:07:51.852742 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.842002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:43891 (local address 127.0.0.1:47592)
0619 15:07:51.844615 (+ 2613us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:51.844634 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:51.844773 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:51.845510 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:51.845515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:51.845542 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:51.846009 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.846029 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.847888 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.847894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.849966 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.849983 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.851963 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.851970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.852148 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.852181 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 15:07:51.852302 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 15:07:51.852445 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2482,"thread_start_us":2328,"threads_started":1}
I20260619 15:07:51.853547 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.840903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.24.103.66:43891)
0619 15:07:51.842501 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:51.842515 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:51.844775 (+ 2260us) server_negotiation.cc:407] Connection header received
0619 15:07:51.844940 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:51.844950 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:51.845262 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:51.845510 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:51.846216 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.847701 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.850180 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.851763 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.851904 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.852115 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:51.853039 (+ 924us) server_negotiation.cc:299] Negotiation successful
0619 15:07:51.853207 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1452,"thread_start_us":1358,"threads_started":1}
I20260619 15:07:51.854681 12792 tablet_service.cc:1467] Processing CreateTablet for tablet 24f9dcc519664e1da75d2a6bc713d8ec (DEFAULT_TABLE table=test-table [id=e668d30ede1d4f19913790f76c4646bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:51.855171 12531 tablet_service.cc:1467] Processing CreateTablet for tablet 24f9dcc519664e1da75d2a6bc713d8ec (DEFAULT_TABLE table=test-table [id=e668d30ede1d4f19913790f76c4646bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:51.856242 12792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24f9dcc519664e1da75d2a6bc713d8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:51.856477 12531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24f9dcc519664e1da75d2a6bc713d8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:51.858871 12662 tablet_service.cc:1467] Processing CreateTablet for tablet 24f9dcc519664e1da75d2a6bc713d8ec (DEFAULT_TABLE table=test-table [id=e668d30ede1d4f19913790f76c4646bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:07:51.860520 12662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24f9dcc519664e1da75d2a6bc713d8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:51.868881 12875 tablet_bootstrap.cc:492] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad: Bootstrap starting.
I20260619 15:07:51.872447 12876 tablet_bootstrap.cc:492] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534: Bootstrap starting.
I20260619 15:07:51.872716 12875 tablet_bootstrap.cc:654] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:51.874207 12875 log.cc:824] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:51.876195 12876 tablet_bootstrap.cc:654] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:51.877574 12876 log.cc:824] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:51.878029 12878 tablet_bootstrap.cc:492] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238: Bootstrap starting.
I20260619 15:07:51.878554 12875 tablet_bootstrap.cc:492] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad: No bootstrap required, opened a new log
I20260619 15:07:51.879010 12875 ts_tablet_manager.cc:1397] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260619 15:07:51.883111 12878 tablet_bootstrap.cc:654] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:51.884451 12878 log.cc:824] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:51.885963 12876 tablet_bootstrap.cc:492] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534: No bootstrap required, opened a new log
I20260619 15:07:51.886297 12876 ts_tablet_manager.cc:1397] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260619 15:07:51.888865 12878 tablet_bootstrap.cc:492] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238: No bootstrap required, opened a new log
I20260619 15:07:51.889173 12878 ts_tablet_manager.cc:1397] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260619 15:07:51.888976 12875 raft_consensus.cc:348] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 33141 } } peers { permanent_uuid: "469f646c24cb4743b04b5aa84592a534" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38623 } } peers { permanent_uuid: "1aad286385914eebbd9cdc1d2bfb6238" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43891 } }
I20260619 15:07:51.889495 12875 raft_consensus.cc:374] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:51.889601 12875 raft_consensus.cc:729] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f18e0e5eaa40bd8f2b84333da902ad, State: Initialized, Role: FOLLOWER
I20260619 15:07:51.890242 12875 consensus_queue.cc:260] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 33141 } } peers { permanent_uuid: "469f646c24cb4743b04b5aa84592a534" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38623 } } peers { permanent_uuid: "1aad286385914eebbd9cdc1d2bfb6238" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43891 } }
I20260619 15:07:51.892325 12875 ts_tablet_manager.cc:1428] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260619 15:07:51.896052 12876 raft_consensus.cc:348] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 33141 } } peers { permanent_uuid: "469f646c24cb4743b04b5aa84592a534" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38623 } } peers { permanent_uuid: "1aad286385914eebbd9cdc1d2bfb6238" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43891 } }
I20260619 15:07:51.896551 12876 raft_consensus.cc:374] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:51.896660 12876 raft_consensus.cc:729] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469f646c24cb4743b04b5aa84592a534, State: Initialized, Role: FOLLOWER
I20260619 15:07:51.897323 12876 consensus_queue.cc:260] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 33141 } } peers { permanent_uuid: "469f646c24cb4743b04b5aa84592a534" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38623 } } peers { permanent_uuid: "1aad286385914eebbd9cdc1d2bfb6238" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43891 } }
I20260619 15:07:51.899739 12857 heartbeater.cc:503] Master 127.24.103.126:38067 was elected leader, sending a full tablet report...
I20260619 15:07:51.900219 12876 ts_tablet_manager.cc:1428] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260619 15:07:51.902021 12878 raft_consensus.cc:348] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 33141 } } peers { permanent_uuid: "469f646c24cb4743b04b5aa84592a534" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38623 } } peers { permanent_uuid: "1aad286385914eebbd9cdc1d2bfb6238" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43891 } }
I20260619 15:07:51.902592 12878 raft_consensus.cc:374] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:51.902712 12878 raft_consensus.cc:729] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aad286385914eebbd9cdc1d2bfb6238, State: Initialized, Role: FOLLOWER
I20260619 15:07:51.903406 12878 consensus_queue.cc:260] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 33141 } } peers { permanent_uuid: "469f646c24cb4743b04b5aa84592a534" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38623 } } peers { permanent_uuid: "1aad286385914eebbd9cdc1d2bfb6238" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43891 } }
I20260619 15:07:51.904655 12727 heartbeater.cc:503] Master 127.24.103.126:38067 was elected leader, sending a full tablet report...
I20260619 15:07:51.905304 12878 ts_tablet_manager.cc:1428] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260619 15:07:51.955726 12882 raft_consensus.cc:482] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:07:51.956075 12882 raft_consensus.cc:504] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 33141 } } peers { permanent_uuid: "469f646c24cb4743b04b5aa84592a534" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38623 } } peers { permanent_uuid: "1aad286385914eebbd9cdc1d2bfb6238" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43891 } }
I20260619 15:07:51.957859 12882 leader_election.cc:288] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5f18e0e5eaa40bd8f2b84333da902ad (127.24.103.65:33141), 1aad286385914eebbd9cdc1d2bfb6238 (127.24.103.66:43891)
I20260619 15:07:51.964661 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.958401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:35367 (local address 127.24.103.65:33141)
0619 15:07:51.958626 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:51.958632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:51.958656 (+ 24us) server_negotiation.cc:407] Connection header received
0619 15:07:51.958738 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:51.958746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:51.958894 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:51.959015 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:51.959675 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.960611 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.962870 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.963666 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.963844 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.963919 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:51.964261 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 15:07:51.964391 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:07:51.964715 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.958234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:33141 (local address 127.24.103.67:35367)
0619 15:07:51.958439 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:51.958460 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:51.958596 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:51.959263 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:51.959268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:51.959298 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:51.959505 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.959519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.960763 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.960768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.962670 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.962683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.963839 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.963845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.964068 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.964131 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 15:07:51.964228 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 15:07:51.964374 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":41}
I20260619 15:07:51.965335 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.959019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:58037 (local address 127.24.103.66:43891)
0619 15:07:51.959226 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:51.959231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:51.959493 (+ 262us) server_negotiation.cc:407] Connection header received
0619 15:07:51.959675 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:51.959680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:51.959790 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:51.959889 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:07:51.960526 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.961508 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.963463 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.964364 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.964533 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.964604 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:51.964937 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 15:07:51.965052 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:07:51.966022 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.958676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:43891 (local address 127.24.103.67:58037)
0619 15:07:51.959384 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 15:07:51.959401 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 15:07:51.959525 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:07:51.959910 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:07:51.959913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:07:51.959931 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:07:51.960373 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.960392 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.961662 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.961673 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.963277 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:07:51.963290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:07:51.964531 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:07:51.964538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:07:51.964716 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:51.964760 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 15:07:51.965527 (+ 767us) client_negotiation.cc:240] Negotiation successful
0619 15:07:51.965665 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":1,"thread_start_us":329,"threads_started":1}
I20260619 15:07:51.965832 12551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f9dcc519664e1da75d2a6bc713d8ec" candidate_uuid: "469f646c24cb4743b04b5aa84592a534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad" is_pre_election: true
I20260619 15:07:51.966449 12551 raft_consensus.cc:2454] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469f646c24cb4743b04b5aa84592a534 in term 0.
I20260619 15:07:51.967159 12746 leader_election.cc:302] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469f646c24cb4743b04b5aa84592a534, c5f18e0e5eaa40bd8f2b84333da902ad; no voters:
I20260619 15:07:51.967144 12682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f9dcc519664e1da75d2a6bc713d8ec" candidate_uuid: "469f646c24cb4743b04b5aa84592a534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aad286385914eebbd9cdc1d2bfb6238" is_pre_election: true
I20260619 15:07:51.967623 12882 raft_consensus.cc:2781] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:07:51.967693 12682 raft_consensus.cc:2454] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469f646c24cb4743b04b5aa84592a534 in term 0.
I20260619 15:07:51.967793 12882 raft_consensus.cc:482] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:07:51.967900 12882 raft_consensus.cc:3037] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:51.970291 12882 raft_consensus.cc:504] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 33141 } } peers { permanent_uuid: "469f646c24cb4743b04b5aa84592a534" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38623 } } peers { permanent_uuid: "1aad286385914eebbd9cdc1d2bfb6238" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43891 } }
I20260619 15:07:51.970913 12882 leader_election.cc:288] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [CANDIDATE]: Term 1 election: Requested vote from peers c5f18e0e5eaa40bd8f2b84333da902ad (127.24.103.65:33141), 1aad286385914eebbd9cdc1d2bfb6238 (127.24.103.66:43891)
I20260619 15:07:51.971458 12682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f9dcc519664e1da75d2a6bc713d8ec" candidate_uuid: "469f646c24cb4743b04b5aa84592a534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aad286385914eebbd9cdc1d2bfb6238"
I20260619 15:07:51.971458 12551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f9dcc519664e1da75d2a6bc713d8ec" candidate_uuid: "469f646c24cb4743b04b5aa84592a534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad"
I20260619 15:07:51.971683 12682 raft_consensus.cc:3037] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:51.971688 12551 raft_consensus.cc:3037] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:51.975102 12682 raft_consensus.cc:2454] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469f646c24cb4743b04b5aa84592a534 in term 1.
I20260619 15:07:51.975102 12551 raft_consensus.cc:2454] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469f646c24cb4743b04b5aa84592a534 in term 1.
I20260619 15:07:51.975567 12746 leader_election.cc:302] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aad286385914eebbd9cdc1d2bfb6238, 469f646c24cb4743b04b5aa84592a534; no voters:
I20260619 15:07:51.975867 12882 raft_consensus.cc:2781] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:51.976325 12882 raft_consensus.cc:686] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 1 LEADER]: Becoming Leader. State: Replica: 469f646c24cb4743b04b5aa84592a534, State: Running, Role: LEADER
I20260619 15:07:51.976845 12882 consensus_queue.cc:237] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 33141 } } peers { permanent_uuid: "469f646c24cb4743b04b5aa84592a534" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38623 } } peers { permanent_uuid: "1aad286385914eebbd9cdc1d2bfb6238" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43891 } }
I20260619 15:07:51.981976 12410 catalog_manager.cc:5495] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 reported cstate change: term changed from 0 to 1, leader changed from <none> to 469f646c24cb4743b04b5aa84592a534 (127.24.103.67). New cstate: current_term: 1 leader_uuid: "469f646c24cb4743b04b5aa84592a534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469f646c24cb4743b04b5aa84592a534" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1aad286385914eebbd9cdc1d2bfb6238" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43891 } health_report { overall_health: UNKNOWN } } }
I20260619 15:07:52.004096 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:51.998655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.24.103.67:38623)
0619 15:07:51.998869 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:51.998875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:51.998895 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:07:51.999145 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:51.999149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:51.999227 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:51.999400 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:07:51.999809 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:52.000530 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:52.001896 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:52.002331 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:52.002472 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:52.003013 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:07:52.003520 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:07:52.003682 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:52.003760 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 15:07:52.003837 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 15:07:52.012604 12682 raft_consensus.cc:1261] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238 [term 1 FOLLOWER]: Refusing update from remote peer 469f646c24cb4743b04b5aa84592a534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:07:52.012616 12551 raft_consensus.cc:1261] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad [term 1 FOLLOWER]: Refusing update from remote peer 469f646c24cb4743b04b5aa84592a534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:07:52.013604 12885 consensus_queue.cc:1035] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f18e0e5eaa40bd8f2b84333da902ad" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:07:52.013960 12882 consensus_queue.cc:1035] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aad286385914eebbd9cdc1d2bfb6238" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 43891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 15:07:52.018126 12858 tablet.cc:2369] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:07:52.024143 12728 tablet.cc:2369] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:07:52.028426 12889 mvcc.cc:205] Tried to move back new op lower bound from 7298587328553041920 to 7298587328422191104. Current Snapshot: MvccSnapshot[applied={T|T < 7298587328553041920}]
I20260619 15:07:52.043932 12410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:07:52.044245 12410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:07:52.048133 12410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28dbe960ad78467e86102ebcb700f965: Sending DeleteTablet for 3 replicas of tablet 24f9dcc519664e1da75d2a6bc713d8ec
I20260619 15:07:52.049317 12792 tablet_service.cc:1514] Processing DeleteTablet for tablet 24f9dcc519664e1da75d2a6bc713d8ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:52 UTC) from {username='slave'} at 127.0.0.1:47960
I20260619 15:07:52.049347 12531 tablet_service.cc:1514] Processing DeleteTablet for tablet 24f9dcc519664e1da75d2a6bc713d8ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:52 UTC) from {username='slave'} at 127.0.0.1:60142
I20260619 15:07:52.050096 12662 tablet_service.cc:1514] Processing DeleteTablet for tablet 24f9dcc519664e1da75d2a6bc713d8ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:07:52 UTC) from {username='slave'} at 127.0.0.1:47592
I20260619 15:07:52.051991 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 12469
I20260619 15:07:52.053124 12898 tablet_replica.cc:330] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad: stopping tablet replica
I20260619 15:07:52.053654 12898 raft_consensus.cc:2229] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:07:52.054239 12898 raft_consensus.cc:2258] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:52.056834 12898 ts_tablet_manager.cc:1905] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:52.067531 12898 ts_tablet_manager.cc:1918] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:52.067746 12898 log.cc:1196] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-0/wal/wals/24f9dcc519664e1da75d2a6bc713d8ec
I20260619 15:07:52.068367 12898 ts_tablet_manager.cc:1939] T 24f9dcc519664e1da75d2a6bc713d8ec P c5f18e0e5eaa40bd8f2b84333da902ad: Deleting consensus metadata
I20260619 15:07:52.071030 12395 catalog_manager.cc:4841] TS c5f18e0e5eaa40bd8f2b84333da902ad (127.24.103.65:33141): tablet 24f9dcc519664e1da75d2a6bc713d8ec (table test-table [id=e668d30ede1d4f19913790f76c4646bb]) successfully deleted
I20260619 15:07:52.072041 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:52.053510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60146 (local address 127.24.103.65:33141)
0619 15:07:52.053751 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:52.053757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:52.061300 (+ 7543us) server_negotiation.cc:407] Connection header received
0619 15:07:52.061874 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:52.061883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:52.062013 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:52.062098 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:52.062979 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:52.064389 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:52.066908 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:52.067761 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:52.067876 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:52.069731 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:52.069746 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:52.069809 (+ 63us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:52.070884 (+ 1075us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:52.070981 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:52.071022 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:52.071137 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:52.071398 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 15:07:52.071552 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 15:07:52.072855 12591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:52.072929 12899 tablet_replica.cc:330] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238: stopping tablet replica
I20260619 15:07:52.073429 12899 raft_consensus.cc:2229] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:07:52.074039 12899 raft_consensus.cc:2258] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:52.074956 12897 tablet_replica.cc:330] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534: stopping tablet replica
I20260619 15:07:52.075423 12897 raft_consensus.cc:2229] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:07:52.076143 12897 raft_consensus.cc:2258] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:07:52.078531 12897 ts_tablet_manager.cc:1905] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:52.080427 12899 ts_tablet_manager.cc:1905] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:07:52.086403 12897 ts_tablet_manager.cc:1918] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:52.086609 12897 log.cc:1196] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-2/wal/wals/24f9dcc519664e1da75d2a6bc713d8ec
I20260619 15:07:52.087262 12897 ts_tablet_manager.cc:1939] T 24f9dcc519664e1da75d2a6bc713d8ec P 469f646c24cb4743b04b5aa84592a534: Deleting consensus metadata
I20260619 15:07:52.089358 12397 catalog_manager.cc:4841] TS 469f646c24cb4743b04b5aa84592a534 (127.24.103.67:38623): tablet 24f9dcc519664e1da75d2a6bc713d8ec (table test-table [id=e668d30ede1d4f19913790f76c4646bb]) successfully deleted
I20260619 15:07:52.096292 12899 ts_tablet_manager.cc:1918] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:07:52.096553 12899 log.cc:1196] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238: Deleting WAL directory at /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881351219607-24989-0/minicluster-data/ts-1/wal/wals/24f9dcc519664e1da75d2a6bc713d8ec
I20260619 15:07:52.097164 12899 ts_tablet_manager.cc:1939] T 24f9dcc519664e1da75d2a6bc713d8ec P 1aad286385914eebbd9cdc1d2bfb6238: Deleting consensus metadata
I20260619 15:07:52.099377 12395 catalog_manager.cc:4841] TS 1aad286385914eebbd9cdc1d2bfb6238 (127.24.103.66:43891): tablet 24f9dcc519664e1da75d2a6bc713d8ec (table test-table [id=e668d30ede1d4f19913790f76c4646bb]) successfully deleted
I20260619 15:07:52.171306 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 12469
I20260619 15:07:52.189018 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 12600
I20260619 15:07:52.197288 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:52.189948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.24.103.66:43891)
0619 15:07:52.190123 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:52.190127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:52.190145 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:07:52.190341 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:52.190345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:52.190425 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:52.190506 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:52.191111 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:52.191864 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:52.193761 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:52.194235 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:52.194339 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:52.195657 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:52.195669 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:52.195689 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:52.196481 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:52.196540 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:52.196563 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:52.196676 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:52.196842 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 15:07:52.196957 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:07:52.198024 12722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:52.297237 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 12600
I20260619 15:07:52.311918 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 12730
I20260619 15:07:52.320096 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:52.312790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.24.103.67:38623)
0619 15:07:52.312988 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:52.312993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:52.313144 (+ 151us) server_negotiation.cc:407] Connection header received
0619 15:07:52.313328 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:52.313332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:52.313429 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:52.313520 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:52.314037 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:52.314921 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:52.316776 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:52.317204 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:52.317305 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:52.318548 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:52.318556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:52.318570 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:52.319291 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:52.319341 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:52.319360 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:52.319439 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:52.319627 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 15:07:52.319720 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:07:52.320873 12852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:52.441933 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 12730
I20260619 15:07:52.461609 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 12379
I20260619 15:07:52.462730 12440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:52.574987 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 12379
2026-06-19T15:07:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2913 ms)
[----------] 6 tests from EncryptionPolicyTest (22830 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:07:52 dist-test-slave-zpbw krb5kdc[12907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:07:52 dist-test-slave-zpbw krb5kdc[12907](info): set up 2 sockets
Jun 19 15:07:52 dist-test-slave-zpbw krb5kdc[12907](info): commencing operation
krb5kdc: starting...
W20260619 15:07:53.731534 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.012s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:07:53 dist-test-slave-zpbw krb5kdc[12907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:07:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:07:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:07:54.001930 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:54.345158 12920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:54.345575 12920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:54.345760 12920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:54.353266 12920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:07:54.353478 12920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:54.353601 12920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:54.353709 12920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:07:54.353803 12920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:07:54.364192 12920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:54.365711 12920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:54.367182 12920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:54.379662 12928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:54.379678 12926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:54.379611 12925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:54.380851 12920 server_base.cc:1034] running on GCE node
I20260619 15:07:54.381788 12920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:54.383464 12920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:54.384758 12920 hybrid_clock.cc:648] HybridClock initialized: now 1781881674384710 us; error 59 us; skew 500 ppm
I20260619 15:07:54.385257 12920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:54 dist-test-slave-zpbw krb5kdc[12907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:54.388863 12920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:07:54.390506 12920 webserver.cc:460] Webserver started at http://127.0.0.1:35723/ using document root <none> and password file <none>
I20260619 15:07:54.391314 12920 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:54.391434 12920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:54.391703 12920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:54.394922 12920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "5f230111dc4943249725bb4784abc010"
format_stamp: "Formatted at 2026-06-19 15:07:54 on dist-test-slave-zpbw"
server_key: "7b921f9ff68071e250ddefdcc3e18a6c"
server_key_iv: "5c87ba72a07b34a6f44f608e90183396"
server_key_version: "encryptionkey@0"
I20260619 15:07:54.395741 12920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "5f230111dc4943249725bb4784abc010"
format_stamp: "Formatted at 2026-06-19 15:07:54 on dist-test-slave-zpbw"
server_key: "7b921f9ff68071e250ddefdcc3e18a6c"
server_key_iv: "5c87ba72a07b34a6f44f608e90183396"
server_key_version: "encryptionkey@0"
I20260619 15:07:54.401325 12920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:07:54.405153 12935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:54.406700 12920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 15:07:54.406982 12920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "5f230111dc4943249725bb4784abc010"
format_stamp: "Formatted at 2026-06-19 15:07:54 on dist-test-slave-zpbw"
server_key: "7b921f9ff68071e250ddefdcc3e18a6c"
server_key_iv: "5c87ba72a07b34a6f44f608e90183396"
server_key_version: "encryptionkey@0"
I20260619 15:07:54.407269 12920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:54.433120 12920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:54.437520 12920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:54.437779 12920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:54.457522 12920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35143
I20260619 15:07:54.457546 12986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35143 every 8 connection(s)
I20260619 15:07:54.459055 12920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:07:54.463166 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:54.469367 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 12920
I20260619 15:07:54.469631 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:07:54.470436 24989 external_mini_cluster.cc:1428] Setting key 51b835b5dcaa5bc87af7c5f6e9cba046
I20260619 15:07:54.475852 12987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010: Bootstrap starting.
I20260619 15:07:54.479317 12987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:54.480466 12987 log.cc:824] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:54.483996 12987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010: No bootstrap required, opened a new log
Jun 19 15:07:54 dist-test-slave-zpbw krb5kdc[12907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 15:07:54.494748 12987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f230111dc4943249725bb4784abc010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35143 } }
I20260619 15:07:54.495326 12987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:54.495436 12987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f230111dc4943249725bb4784abc010, State: Initialized, Role: FOLLOWER
I20260619 15:07:54.496150 12987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f230111dc4943249725bb4784abc010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35143 } }
I20260619 15:07:54.496451 12987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:07:54.496594 12987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:07:54.496681 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:54.472817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44621 (local address 127.0.0.1:35143)
0619 15:07:54.473732 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:54.473745 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:54.473790 (+ 45us) server_negotiation.cc:407] Connection header received
0619 15:07:54.475703 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:54.475737 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:54.476251 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:54.476663 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:54.478236 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:54.479752 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:54.481276 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:54.482177 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:54.482388 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:54.486850 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:54.486863 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:54.486893 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:54.486952 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:54.491396 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:54.492424 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:54.492434 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:54.492444 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:54.492698 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:54.493174 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:54.493180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:54.493184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:54.493983 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:54.494301 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:54.494710 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 15:07:54.495148 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":395,"threads_started":1}
I20260619 15:07:54.496803 12987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:54.499480 12987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f230111dc4943249725bb4784abc010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35143 } }
I20260619 15:07:54.500125 12987 leader_election.cc:302] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f230111dc4943249725bb4784abc010; no voters:
I20260619 15:07:54.500842 12987 leader_election.cc:288] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:07:54.500963 12992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:54.501310 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:07:54.503645 12987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:07:54.506230 12992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [term 1 LEADER]: Becoming Leader. State: Replica: 5f230111dc4943249725bb4784abc010, State: Running, Role: LEADER
I20260619 15:07:54.507187 12992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f230111dc4943249725bb4784abc010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35143 } }
I20260619 15:07:54.510964 12993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f230111dc4943249725bb4784abc010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f230111dc4943249725bb4784abc010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35143 } } }
I20260619 15:07:54.511492 12993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:54.512187 12994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f230111dc4943249725bb4784abc010. Latest consensus state: current_term: 1 leader_uuid: "5f230111dc4943249725bb4784abc010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f230111dc4943249725bb4784abc010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35143 } } }
I20260619 15:07:54.512523 12994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:54.512429 13003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:07:54.528306 13003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:07:54.532706 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:54.507135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54975 (local address 127.0.0.1:35143)
0619 15:07:54.507320 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:54.507326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:54.507347 (+ 21us) server_negotiation.cc:407] Connection header received
0619 15:07:54.507511 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:54.507517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:54.507614 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:54.507675 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:54.509470 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:54.510670 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:54.512579 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:54.513582 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:54.513711 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:54.518286 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:54.518297 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:54.518305 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:07:54.518372 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:54.527415 (+ 9043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:54.529075 (+ 1660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:54.529084 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:54.529091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:54.529332 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:07:54.530153 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:07:54.530162 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:07:54.530167 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:07:54.530763 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:54.531152 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:54.531972 (+ 820us) server_negotiation.cc:299] Negotiation successful
0619 15:07:54.532251 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 15:07:54.541199 13003 catalog_manager.cc:1347] Generated new cluster ID: a4c5552b2d204fd99bb0bcddcf01eb78
I20260619 15:07:54.541353 13003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:07:54.558347 13003 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:07:54.559823 13003 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:07:54.569386 13003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f230111dc4943249725bb4784abc010: Generated new TSK 0
I20260619 15:07:54.570370 13003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:07:54.586059 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 12920
I20260619 15:07:54.587560 12982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:07:54.681310 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 12920
W20260619 15:07:54.699040 12905 connection.cc:535] client connection to 127.0.0.1:35143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35143): Connection reset by peer (error 104)
2026-06-19T15:07:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2101 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:07:54 dist-test-slave-zpbw krb5kdc[13020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:07:54 dist-test-slave-zpbw krb5kdc[13020](info): set up 2 sockets
Jun 19 15:07:54 dist-test-slave-zpbw krb5kdc[13020](info): commencing operation
krb5kdc: starting...
W20260619 15:07:55.795327 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:07:55 dist-test-slave-zpbw krb5kdc[13020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:07:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:07:56.058246 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:56.399042 13033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:56.399367 13033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:56.399464 13033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:56.407331 13033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:07:56.407465 13033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:56.407512 13033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:56.407574 13033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:07:56.407621 13033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:07:56.418208 13033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:56.419560 13033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:56.420890 13033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:56.433519 13039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:56.433737 13038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:56.433719 13041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:56.439858 13033 server_base.cc:1034] running on GCE node
I20260619 15:07:56.440846 13033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:56.442430 13033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:56.443671 13033 hybrid_clock.cc:648] HybridClock initialized: now 1781881676443629 us; error 50 us; skew 500 ppm
I20260619 15:07:56.444049 13033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:56 dist-test-slave-zpbw krb5kdc[13020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:56.452231 13033 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:07:56.453980 13033 webserver.cc:460] Webserver started at http://127.0.0.1:46661/ using document root <none> and password file <none>
I20260619 15:07:56.454764 13033 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:56.455024 13033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:56.455351 13033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:56.458463 13033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "0c5a6293173d45e0a9353747d1800a5b"
format_stamp: "Formatted at 2026-06-19 15:07:56 on dist-test-slave-zpbw"
server_key: "f0bb63d1dd1e027bed444e34d97b5e72"
server_key_iv: "51b205f221c96b501530e8fb7de85cbb"
server_key_version: "encryptionkey@0"
I20260619 15:07:56.459314 13033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "0c5a6293173d45e0a9353747d1800a5b"
format_stamp: "Formatted at 2026-06-19 15:07:56 on dist-test-slave-zpbw"
server_key: "f0bb63d1dd1e027bed444e34d97b5e72"
server_key_iv: "51b205f221c96b501530e8fb7de85cbb"
server_key_version: "encryptionkey@0"
I20260619 15:07:56.464838 13033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:07:56.468689 13048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:56.470103 13033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:07:56.470338 13033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "0c5a6293173d45e0a9353747d1800a5b"
format_stamp: "Formatted at 2026-06-19 15:07:56 on dist-test-slave-zpbw"
server_key: "f0bb63d1dd1e027bed444e34d97b5e72"
server_key_iv: "51b205f221c96b501530e8fb7de85cbb"
server_key_version: "encryptionkey@0"
I20260619 15:07:56.470566 13033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:56.535916 13033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:56.540728 13033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:56.541061 13033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:56.560613 13033 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35479
I20260619 15:07:56.560647 13099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35479 every 8 connection(s)
I20260619 15:07:56.562111 13033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:07:56.565847 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 13033
I20260619 15:07:56.566061 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:07:56.566082 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:56.566509 24989 external_mini_cluster.cc:1428] Setting key da9149fbf7342851c76e641ef3517458
I20260619 15:07:56.580812 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:56.568444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35791 (local address 127.0.0.1:35479)
0619 15:07:56.569325 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:56.569337 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:56.569383 (+ 46us) server_negotiation.cc:407] Connection header received
0619 15:07:56.571064 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:56.571090 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:56.571559 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:56.571923 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:56.572565 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:56.574014 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:56.575949 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:56.576751 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:56.576907 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:56.578335 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:56.578344 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:56.578359 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:56.579021 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:56.579112 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:56.579141 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:56.579275 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:56.579585 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 15:07:56.579786 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":424,"threads_started":1}
I20260619 15:07:56.581061 13100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b: Bootstrap starting.
I20260619 15:07:56.584535 13100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:56.585099 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:07:56.585911 13100 log.cc:824] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:56.589568 13100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b: No bootstrap required, opened a new log
I20260619 15:07:56.597692 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:56.590402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56603 (local address 127.0.0.1:35479)
0619 15:07:56.590599 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:56.590605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:56.591060 (+ 455us) server_negotiation.cc:407] Connection header received
0619 15:07:56.591230 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:56.591235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:56.591319 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:56.591439 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:56.591937 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:56.593218 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:56.594634 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:07:56.595450 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:07:56.595586 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:07:56.596787 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:56.596794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:56.596798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:56.596821 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:56.596893 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:56.596914 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:56.597066 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:56.597251 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 15:07:56.597312 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:07:56.599861 13100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5a6293173d45e0a9353747d1800a5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35479 } }
I20260619 15:07:56.600256 13100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:56.600337 13100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c5a6293173d45e0a9353747d1800a5b, State: Initialized, Role: FOLLOWER
I20260619 15:07:56.600940 13100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5a6293173d45e0a9353747d1800a5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35479 } }
I20260619 15:07:56.601150 13100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:07:56.601281 13100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:07:56.601408 13100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:56.603736 13100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5a6293173d45e0a9353747d1800a5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35479 } }
I20260619 15:07:56.604321 13100 leader_election.cc:302] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c5a6293173d45e0a9353747d1800a5b; no voters:
I20260619 15:07:56.604955 13100 leader_election.cc:288] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:07:56.605118 13110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:56.605835 13110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [term 1 LEADER]: Becoming Leader. State: Replica: 0c5a6293173d45e0a9353747d1800a5b, State: Running, Role: LEADER
I20260619 15:07:56.606439 13110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5a6293173d45e0a9353747d1800a5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35479 } }
I20260619 15:07:56.607339 13100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:07:56.609265 13111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c5a6293173d45e0a9353747d1800a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5a6293173d45e0a9353747d1800a5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35479 } } }
I20260619 15:07:56.609664 13111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:56.610565 13115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:07:56.610414 13112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c5a6293173d45e0a9353747d1800a5b. Latest consensus state: current_term: 1 leader_uuid: "0c5a6293173d45e0a9353747d1800a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5a6293173d45e0a9353747d1800a5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35479 } } }
I20260619 15:07:56.611338 13112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:56.615867 13115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:07:56.626817 13115 catalog_manager.cc:1347] Generated new cluster ID: 12532fdba61c4c67912258fb477f4577
I20260619 15:07:56.626956 13115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:07:56.656466 13115 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:07:56.658466 13115 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:07:56.665812 13115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c5a6293173d45e0a9353747d1800a5b: Generated new TSK 0
I20260619 15:07:56.667088 13115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:07:56.682396 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13033
W20260619 15:07:56.684504 13095 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35791
W20260619 15:07:56.685407 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13033: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:07:56.685606 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 13033
W20260619 15:07:56.702461 13018 connection.cc:535] client connection to 127.0.0.1:35479 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35479): Connection reset by peer (error 104)
2026-06-19T15:07:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:07:56 dist-test-slave-zpbw krb5kdc[13131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:07:56 dist-test-slave-zpbw krb5kdc[13131](info): set up 2 sockets
Jun 19 15:07:56 dist-test-slave-zpbw krb5kdc[13131](info): commencing operation
krb5kdc: starting...
W20260619 15:07:57.823411 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:07:57 dist-test-slave-zpbw krb5kdc[13131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:07:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:07:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:07:58.086493 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:07:58.427065 13144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:07:58.427404 13144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:07:58.427595 13144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:07:58.435107 13144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:07:58.435237 13144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:07:58.435336 13144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:07:58.435451 13144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:07:58.435534 13144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:07:58.446218 13144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46515
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:07:58.447696 13144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:07:58.449100 13144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:07:58.461493 13150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:58.461454 13149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:07:58.461540 13152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:07:58.462467 13144 server_base.cc:1034] running on GCE node
I20260619 15:07:58.463430 13144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:07:58.464996 13144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:07:58.466254 13144 hybrid_clock.cc:648] HybridClock initialized: now 1781881678466210 us; error 64 us; skew 500 ppm
I20260619 15:07:58.466662 13144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:07:58 dist-test-slave-zpbw krb5kdc[13131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:07:58.471552 13144 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:07:58.477787 13144 webserver.cc:460] Webserver started at http://127.0.0.1:41703/ using document root <none> and password file <none>
I20260619 15:07:58.478993 13144 fs_manager.cc:359] Metadata directory not provided
I20260619 15:07:58.479213 13144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:07:58.479689 13144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:07:58.484807 13144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "b4d1a418e9724cf8a2dd74c7a2c0b143"
format_stamp: "Formatted at 2026-06-19 15:07:58 on dist-test-slave-zpbw"
server_key: "dade942478a68903ca1ed47478b5dba1"
server_key_iv: "b90fae2341ed6c2c10a4039e106baffa"
server_key_version: "encryptionkey@0"
I20260619 15:07:58.486083 13144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "b4d1a418e9724cf8a2dd74c7a2c0b143"
format_stamp: "Formatted at 2026-06-19 15:07:58 on dist-test-slave-zpbw"
server_key: "dade942478a68903ca1ed47478b5dba1"
server_key_iv: "b90fae2341ed6c2c10a4039e106baffa"
server_key_version: "encryptionkey@0"
I20260619 15:07:58.495358 13144 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260619 15:07:58.502198 13159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:07:58.504105 13144 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 15:07:58.504339 13144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "b4d1a418e9724cf8a2dd74c7a2c0b143"
format_stamp: "Formatted at 2026-06-19 15:07:58 on dist-test-slave-zpbw"
server_key: "dade942478a68903ca1ed47478b5dba1"
server_key_iv: "b90fae2341ed6c2c10a4039e106baffa"
server_key_version: "encryptionkey@0"
I20260619 15:07:58.504581 13144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:07:58.511965 13144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:07:58.517053 13144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:07:58.517406 13144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:07:58.537485 13144 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46515
I20260619 15:07:58.537554 13210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46515 every 8 connection(s)
I20260619 15:07:58.539474 13144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:07:58.543119 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 13144
I20260619 15:07:58.543326 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:07:58.543867 24989 external_mini_cluster.cc:1428] Setting key f0f4be0e528ca329e034fe5e529ff18b
I20260619 15:07:58.545181 13211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:07:58.553778 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:58.545946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33185 (local address 127.0.0.1:46515)
0619 15:07:58.546932 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:58.546948 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:58.547005 (+ 57us) server_negotiation.cc:407] Connection header received
0619 15:07:58.549248 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:58.549282 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:58.549819 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:58.550356 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:58.551544 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:58.551553 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:58.551571 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:58.551596 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:58.551700 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:58.551731 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:58.551986 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:58.552125 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 15:07:58.552371 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":397,"threads_started":1}
I20260619 15:07:58.557945 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:07:58.560497 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143: Bootstrap starting.
I20260619 15:07:58.564788 13211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143: Neither blocks nor log segments found. Creating new log.
I20260619 15:07:58.565971 13211 log.cc:824] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143: Log is configured to *not* fsync() on all Append() calls
I20260619 15:07:58.567114 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:07:58.562654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48639 (local address 127.0.0.1:46515)
0619 15:07:58.562895 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 15:07:58.562899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:07:58.564898 (+ 1999us) server_negotiation.cc:407] Connection header received
0619 15:07:58.565116 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:07:58.565120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:07:58.565177 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:07:58.565299 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:07:58.566261 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:07:58.566267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:07:58.566272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:07:58.566290 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:07:58.566337 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:07:58.566358 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:07:58.566499 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 15:07:58.566640 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 15:07:58.566709 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 15:07:58.569144 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143: No bootstrap required, opened a new log
I20260619 15:07:58.576699 13211 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d1a418e9724cf8a2dd74c7a2c0b143" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46515 } }
I20260619 15:07:58.577150 13211 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:07:58.577289 13211 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d1a418e9724cf8a2dd74c7a2c0b143, State: Initialized, Role: FOLLOWER
I20260619 15:07:58.577921 13211 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d1a418e9724cf8a2dd74c7a2c0b143" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46515 } }
I20260619 15:07:58.578166 13211 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:07:58.578346 13211 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:07:58.578534 13211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:07:58.580996 13211 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d1a418e9724cf8a2dd74c7a2c0b143" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46515 } }
I20260619 15:07:58.581578 13211 leader_election.cc:302] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4d1a418e9724cf8a2dd74c7a2c0b143; no voters:
I20260619 15:07:58.582192 13211 leader_election.cc:288] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:07:58.582350 13221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:07:58.583079 13221 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [term 1 LEADER]: Becoming Leader. State: Replica: b4d1a418e9724cf8a2dd74c7a2c0b143, State: Running, Role: LEADER
I20260619 15:07:58.583688 13221 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d1a418e9724cf8a2dd74c7a2c0b143" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46515 } }
I20260619 15:07:58.584442 13211 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:07:58.586988 13222 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4d1a418e9724cf8a2dd74c7a2c0b143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d1a418e9724cf8a2dd74c7a2c0b143" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46515 } } }
I20260619 15:07:58.587560 13222 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:58.587083 13223 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4d1a418e9724cf8a2dd74c7a2c0b143. Latest consensus state: current_term: 1 leader_uuid: "b4d1a418e9724cf8a2dd74c7a2c0b143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d1a418e9724cf8a2dd74c7a2c0b143" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46515 } } }
I20260619 15:07:58.588515 13223 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143 [sys.catalog]: This master's current role is: LEADER
I20260619 15:07:58.588624 13230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:07:58.594586 13230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:07:58.604163 13230 catalog_manager.cc:1347] Generated new cluster ID: 037d2de3fba24380822379e51c217259
I20260619 15:07:58.604349 13230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:07:58.625708 13230 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:07:58.626174 13230 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:07:58.632007 13230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4d1a418e9724cf8a2dd74c7a2c0b143: Generated new TSK 0
I20260619 15:07:58.633246 13230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:07:58.650153 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13144
W20260619 15:07:58.651480 13206 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33185
W20260619 15:07:58.652298 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13144: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:07:58.652478 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 13144
2026-06-19T15:07:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:07:58 dist-test-slave-zpbw krb5kdc[13242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:07:58 dist-test-slave-zpbw krb5kdc[13242](info): set up 2 sockets
Jun 19 15:07:58 dist-test-slave-zpbw krb5kdc[13242](info): commencing operation
krb5kdc: starting...
W20260619 15:07:59.767501 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:07:59 dist-test-slave-zpbw krb5kdc[13242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:07:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:08:00.032114 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:00.372562 13255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:00.372874 13255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:00.372974 13255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:00.380411 13255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:00.380544 13255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:00.380636 13255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:00.380725 13255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:00.380786 13255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:00.390956 13255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:00.392318 13255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:00.393688 13255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:00.405804 13260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:00.405719 13261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:00.405778 13263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:00.406664 13255 server_base.cc:1034] running on GCE node
I20260619 15:08:00.407588 13255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:00.409044 13255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:00.410297 13255 hybrid_clock.cc:648] HybridClock initialized: now 1781881680410241 us; error 43 us; skew 500 ppm
I20260619 15:08:00.410640 13255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:00 dist-test-slave-zpbw krb5kdc[13242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:00.414040 13255 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:08:00.415652 13255 webserver.cc:460] Webserver started at http://127.0.0.1:37777/ using document root <none> and password file <none>
I20260619 15:08:00.416318 13255 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:00.416420 13255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:00.416677 13255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:00.419796 13255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "71a8682fd4af4c57bdc7bb42dc19bbcc"
format_stamp: "Formatted at 2026-06-19 15:08:00 on dist-test-slave-zpbw"
server_key: "cb850cd738847dfd912b0320eda8ba4e"
server_key_iv: "817098a1069bb13e3d9f1a8bd166411e"
server_key_version: "encryptionkey@0"
I20260619 15:08:00.420547 13255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "71a8682fd4af4c57bdc7bb42dc19bbcc"
format_stamp: "Formatted at 2026-06-19 15:08:00 on dist-test-slave-zpbw"
server_key: "cb850cd738847dfd912b0320eda8ba4e"
server_key_iv: "817098a1069bb13e3d9f1a8bd166411e"
server_key_version: "encryptionkey@0"
I20260619 15:08:00.426062 13255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:08:00.429659 13270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:00.431192 13255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:08:00.431409 13255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "71a8682fd4af4c57bdc7bb42dc19bbcc"
format_stamp: "Formatted at 2026-06-19 15:08:00 on dist-test-slave-zpbw"
server_key: "cb850cd738847dfd912b0320eda8ba4e"
server_key_iv: "817098a1069bb13e3d9f1a8bd166411e"
server_key_version: "encryptionkey@0"
I20260619 15:08:00.431623 13255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:00.450726 13255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:00.454988 13255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:00.455288 13255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:00.474086 13255 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37483
I20260619 15:08:00.474112 13321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37483 every 8 connection(s)
I20260619 15:08:00.475592 13255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:00.478916 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 13255
I20260619 15:08:00.479115 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:00.479570 24989 external_mini_cluster.cc:1428] Setting key e1af26fd12ae57d7bb01290ac7829064
I20260619 15:08:00.479696 13322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:00.495301 13322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc: Bootstrap starting.
Jun 19 15:08:00 dist-test-slave-zpbw krb5kdc[13242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 15:08:00.499161 13322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:00.500344 13322 log.cc:824] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:00.503490 13322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc: No bootstrap required, opened a new log
I20260619 15:08:00.504863 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:00.481388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38743 (local address 127.0.0.1:37483)
0619 15:08:00.482222 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:00.482233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:00.482281 (+ 48us) server_negotiation.cc:407] Connection header received
0619 15:08:00.484387 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:00.484409 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:00.484864 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:00.485211 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:00.487121 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:00.488506 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:00.490459 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:00.491326 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:00.491481 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:00.496906 (+ 5425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:00.496916 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:00.496931 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:00.496975 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:00.500320 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:00.501240 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:00.501250 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:00.501259 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:00.501455 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:00.501891 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:00.501897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:00.501900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:00.502470 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:00.502774 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:00.503323 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 15:08:00.503801 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":428,"threads_started":1}
I20260619 15:08:00.508906 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:00.513432 13322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a8682fd4af4c57bdc7bb42dc19bbcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260619 15:08:00.513984 13322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:00.514108 13322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a8682fd4af4c57bdc7bb42dc19bbcc, State: Initialized, Role: FOLLOWER
I20260619 15:08:00.514962 13322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a8682fd4af4c57bdc7bb42dc19bbcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260619 15:08:00.515384 13322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:00.515555 13322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:00.515779 13322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:00.519836 13322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a8682fd4af4c57bdc7bb42dc19bbcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260619 15:08:00.520452 13322 leader_election.cc:302] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71a8682fd4af4c57bdc7bb42dc19bbcc; no voters:
I20260619 15:08:00.521100 13322 leader_election.cc:288] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:00.521266 13332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:00.522445 13332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [term 1 LEADER]: Becoming Leader. State: Replica: 71a8682fd4af4c57bdc7bb42dc19bbcc, State: Running, Role: LEADER
I20260619 15:08:00.523242 13332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a8682fd4af4c57bdc7bb42dc19bbcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260619 15:08:00.523716 13322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:00.526338 13333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71a8682fd4af4c57bdc7bb42dc19bbcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a8682fd4af4c57bdc7bb42dc19bbcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } } }
I20260619 15:08:00.526840 13333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:00.530678 13334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71a8682fd4af4c57bdc7bb42dc19bbcc. Latest consensus state: current_term: 1 leader_uuid: "71a8682fd4af4c57bdc7bb42dc19bbcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a8682fd4af4c57bdc7bb42dc19bbcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } } }
I20260619 15:08:00.531811 13334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:00.535339 13337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:00.538955 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:00.514750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35309 (local address 127.0.0.1:37483)
0619 15:08:00.514986 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:00.514991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:00.519401 (+ 4410us) server_negotiation.cc:407] Connection header received
0619 15:08:00.519619 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:00.519625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:00.519738 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:00.519886 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:00.521918 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:00.523225 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:00.525647 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:00.526392 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:00.526506 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:00.530588 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:00.530595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:00.530600 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:00.530647 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:00.535055 (+ 4408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:00.536079 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:00.536085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:00.536088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:00.536304 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:00.536823 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:00.536828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:00.536831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:00.537254 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:00.537568 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:00.538074 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 15:08:00.538390 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 15:08:00.542464 13337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:00.551971 13337 catalog_manager.cc:1347] Generated new cluster ID: 38d10c88ef234db4850692ccd455349c
I20260619 15:08:00.552117 13337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:00.569979 13337 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:00.572048 13337 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:00.584223 13337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71a8682fd4af4c57bdc7bb42dc19bbcc: Generated new TSK 0
I20260619 15:08:00.585503 13337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:00.594969 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13255
I20260619 15:08:00.596127 13317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:08:00.688498 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 13255
2026-06-19T15:08:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2035 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:00 dist-test-slave-zpbw krb5kdc[13354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:00 dist-test-slave-zpbw krb5kdc[13354](info): set up 2 sockets
Jun 19 15:08:00 dist-test-slave-zpbw krb5kdc[13354](info): commencing operation
krb5kdc: starting...
W20260619 15:08:01.819559 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:01 dist-test-slave-zpbw krb5kdc[13354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:08:02.089500 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:02.436378 13367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:02.436769 13367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:02.436932 13367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:02.444788 13367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:02.444968 13367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:02.445078 13367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:02.445188 13367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:02.445269 13367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:02.455816 13367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:02.457258 13367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:02.458647 13367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:02.471025 13373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:02.471063 13372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:02.471138 13375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:02.471967 13367 server_base.cc:1034] running on GCE node
I20260619 15:08:02.472785 13367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:02.474233 13367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:02.475435 13367 hybrid_clock.cc:648] HybridClock initialized: now 1781881682475402 us; error 39 us; skew 500 ppm
I20260619 15:08:02.475802 13367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:02 dist-test-slave-zpbw krb5kdc[13354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:02.483191 13367 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:08:02.484932 13367 webserver.cc:460] Webserver started at http://127.0.0.1:37897/ using document root <none> and password file <none>
I20260619 15:08:02.485729 13367 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:02.485839 13367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:02.486109 13367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:02.489344 13367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "29c64af7683a4b47a74cb3d7ebf708e7"
format_stamp: "Formatted at 2026-06-19 15:08:02 on dist-test-slave-zpbw"
server_key: "940b86be7775209b70f1594ca0f1e967"
server_key_iv: "7ad8a33d63711cb22ff883df2e534ad5"
server_key_version: "encryptionkey@0"
I20260619 15:08:02.490118 13367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "29c64af7683a4b47a74cb3d7ebf708e7"
format_stamp: "Formatted at 2026-06-19 15:08:02 on dist-test-slave-zpbw"
server_key: "940b86be7775209b70f1594ca0f1e967"
server_key_iv: "7ad8a33d63711cb22ff883df2e534ad5"
server_key_version: "encryptionkey@0"
I20260619 15:08:02.495890 13367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:08:02.499810 13382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:02.501361 13367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:08:02.501554 13367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "29c64af7683a4b47a74cb3d7ebf708e7"
format_stamp: "Formatted at 2026-06-19 15:08:02 on dist-test-slave-zpbw"
server_key: "940b86be7775209b70f1594ca0f1e967"
server_key_iv: "7ad8a33d63711cb22ff883df2e534ad5"
server_key_version: "encryptionkey@0"
I20260619 15:08:02.501786 13367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:02.520802 13367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:02.525362 13367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:02.525669 13367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:02.545172 13367 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38269
I20260619 15:08:02.545195 13433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38269 every 8 connection(s)
I20260619 15:08:02.546695 13367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:02.550750 13434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:02.556756 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 13367
I20260619 15:08:02.556952 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:02.557509 24989 external_mini_cluster.cc:1428] Setting key be21ac945d5f0ab15adb73668adbc34d
I20260619 15:08:02.562863 13434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7: Bootstrap starting.
I20260619 15:08:02.566828 13434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:02.568085 13434 log.cc:824] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:02.571519 13434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7: No bootstrap required, opened a new log
I20260619 15:08:02.572094 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:02.559520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47323 (local address 127.0.0.1:38269)
0619 15:08:02.560535 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:02.560548 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:02.560599 (+ 51us) server_negotiation.cc:407] Connection header received
0619 15:08:02.562371 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:02.562398 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:02.562930 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:02.563353 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:02.564021 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:02.565392 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:02.567388 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:02.568088 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:02.568258 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:02.569603 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:02.569622 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:02.569638 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:02.570209 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:02.570287 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:02.570312 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:02.570405 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:02.570655 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 15:08:02.570977 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":477,"threads_started":1}
I20260619 15:08:02.576498 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:02.580530 13434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c64af7683a4b47a74cb3d7ebf708e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38269 } }
I20260619 15:08:02.581116 13434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:02.581269 13434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c64af7683a4b47a74cb3d7ebf708e7, State: Initialized, Role: FOLLOWER
I20260619 15:08:02.582093 13434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c64af7683a4b47a74cb3d7ebf708e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38269 } }
I20260619 15:08:02.582444 13434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:02.582605 13434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:02.582901 13434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:02.586140 13434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c64af7683a4b47a74cb3d7ebf708e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38269 } }
I20260619 15:08:02.586864 13434 leader_election.cc:302] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29c64af7683a4b47a74cb3d7ebf708e7; no voters:
I20260619 15:08:02.587702 13434 leader_election.cc:288] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:02.587831 13444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:02.588246 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:02.581240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35281 (local address 127.0.0.1:38269)
0619 15:08:02.581466 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:02.581472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:02.581790 (+ 318us) server_negotiation.cc:407] Connection header received
0619 15:08:02.581980 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:02.581984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:02.582045 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:02.582132 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:02.582849 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:02.583640 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:02.585545 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:02.585988 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:02.586082 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:02.587315 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:02.587319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:02.587323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:02.587342 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:02.587381 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:02.587396 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:02.587516 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:02.587663 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 15:08:02.587772 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 15:08:02.589182 13444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [term 1 LEADER]: Becoming Leader. State: Replica: 29c64af7683a4b47a74cb3d7ebf708e7, State: Running, Role: LEADER
I20260619 15:08:02.589972 13444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c64af7683a4b47a74cb3d7ebf708e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38269 } }
I20260619 15:08:02.590502 13434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:02.593046 13446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c64af7683a4b47a74cb3d7ebf708e7. Latest consensus state: current_term: 1 leader_uuid: "29c64af7683a4b47a74cb3d7ebf708e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c64af7683a4b47a74cb3d7ebf708e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38269 } } }
I20260619 15:08:02.593544 13446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:02.594241 13445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29c64af7683a4b47a74cb3d7ebf708e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c64af7683a4b47a74cb3d7ebf708e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38269 } } }
I20260619 15:08:02.594612 13445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:02.594666 13453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:02.607133 13453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:02.617671 13453 catalog_manager.cc:1347] Generated new cluster ID: 664d43f6bac741b7a7dc9e708a6562eb
I20260619 15:08:02.617816 13453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:02.631166 13453 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:02.633225 13453 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:02.645243 13453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29c64af7683a4b47a74cb3d7ebf708e7: Generated new TSK 0
I20260619 15:08:02.646214 13453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:02.660948 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13367
W20260619 15:08:02.662104 13429 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47323
W20260619 15:08:02.662971 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13367: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:08:02.663134 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 13367
2026-06-19T15:08:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:02 dist-test-slave-zpbw krb5kdc[13466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:02 dist-test-slave-zpbw krb5kdc[13466](info): set up 2 sockets
Jun 19 15:08:02 dist-test-slave-zpbw krb5kdc[13466](info): commencing operation
krb5kdc: starting...
W20260619 15:08:03.775616 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:03 dist-test-slave-zpbw krb5kdc[13466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:08:04.041407 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:04.383998 13479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:04.384339 13479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:04.384438 13479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:04.392042 13479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:04.392176 13479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:04.392230 13479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:04.392284 13479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:04.392320 13479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:04.403605 13479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:04.404950 13479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:04.406287 13479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:04.419271 13487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:04.419364 13484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:04.419340 13485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:04.420158 13479 server_base.cc:1034] running on GCE node
I20260619 15:08:04.420948 13479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:04.422425 13479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:04.423638 13479 hybrid_clock.cc:648] HybridClock initialized: now 1781881684423581 us; error 46 us; skew 500 ppm
I20260619 15:08:04.424027 13479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:04 dist-test-slave-zpbw krb5kdc[13466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:04.427380 13479 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:08:04.428980 13479 webserver.cc:460] Webserver started at http://127.0.0.1:46839/ using document root <none> and password file <none>
I20260619 15:08:04.429682 13479 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:04.429788 13479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:04.430056 13479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:04.433375 13479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "e4dcebe747914c8f9fbffbfb3674c8e5"
format_stamp: "Formatted at 2026-06-19 15:08:04 on dist-test-slave-zpbw"
server_key: "c0d1a5350a125aea7c0808ab6ca87265"
server_key_iv: "373e52f50ef099302159341e1a5d8b46"
server_key_version: "encryptionkey@0"
I20260619 15:08:04.434181 13479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "e4dcebe747914c8f9fbffbfb3674c8e5"
format_stamp: "Formatted at 2026-06-19 15:08:04 on dist-test-slave-zpbw"
server_key: "c0d1a5350a125aea7c0808ab6ca87265"
server_key_iv: "373e52f50ef099302159341e1a5d8b46"
server_key_version: "encryptionkey@0"
I20260619 15:08:04.440038 13479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 15:08:04.443866 13494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:04.445402 13479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:08:04.445564 13479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "e4dcebe747914c8f9fbffbfb3674c8e5"
format_stamp: "Formatted at 2026-06-19 15:08:04 on dist-test-slave-zpbw"
server_key: "c0d1a5350a125aea7c0808ab6ca87265"
server_key_iv: "373e52f50ef099302159341e1a5d8b46"
server_key_version: "encryptionkey@0"
I20260619 15:08:04.445782 13479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:04.452248 13479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:04.456591 13479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:04.456871 13479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:04.476517 13479 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42453
I20260619 15:08:04.476560 13545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42453 every 8 connection(s)
I20260619 15:08:04.478154 13479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:04.482072 13546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:04.488344 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 13479
I20260619 15:08:04.488533 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:04.488987 24989 external_mini_cluster.cc:1428] Setting key eafb8f1f203870c0562222814682584f
I20260619 15:08:04.493587 13546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5: Bootstrap starting.
I20260619 15:08:04.497376 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:04.491080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37235 (local address 127.0.0.1:42453)
0619 15:08:04.491989 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:04.492002 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:04.492044 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:08:04.493800 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:04.493837 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:04.494196 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:04.494535 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:04.495757 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:04.495764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:04.495777 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:04.495791 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:04.495850 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:04.495870 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:04.496044 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:04.496192 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 15:08:04.496416 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":424,"threads_started":1}
I20260619 15:08:04.497484 13546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:04.498929 13546 log.cc:824] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:04.501758 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:04.502324 13546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5: No bootstrap required, opened a new log
I20260619 15:08:04.509696 13546 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dcebe747914c8f9fbffbfb3674c8e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42453 } }
I20260619 15:08:04.510099 13546 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:04.510311 13546 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4dcebe747914c8f9fbffbfb3674c8e5, State: Initialized, Role: FOLLOWER
I20260619 15:08:04.511068 13546 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dcebe747914c8f9fbffbfb3674c8e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42453 } }
I20260619 15:08:04.511356 13546 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:04.511518 13546 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:04.511695 13546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:04.515071 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:04.512511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57899 (local address 127.0.0.1:42453)
0619 15:08:04.512707 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:04.512727 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:04.513040 (+ 313us) server_negotiation.cc:407] Connection header received
0619 15:08:04.513195 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:04.513202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:04.513281 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:04.513406 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:04.514326 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:04.514332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:04.514337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:04.514355 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:04.514401 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:04.514420 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:04.514547 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:04.514613 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 15:08:04.514673 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 15:08:04.515074 13546 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dcebe747914c8f9fbffbfb3674c8e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42453 } }
I20260619 15:08:04.515786 13546 leader_election.cc:302] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4dcebe747914c8f9fbffbfb3674c8e5; no voters:
I20260619 15:08:04.516476 13546 leader_election.cc:288] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:04.516602 13556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:04.517400 13556 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [term 1 LEADER]: Becoming Leader. State: Replica: e4dcebe747914c8f9fbffbfb3674c8e5, State: Running, Role: LEADER
I20260619 15:08:04.518026 13556 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dcebe747914c8f9fbffbfb3674c8e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42453 } }
I20260619 15:08:04.518818 13546 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:04.520721 13557 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4dcebe747914c8f9fbffbfb3674c8e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dcebe747914c8f9fbffbfb3674c8e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42453 } } }
I20260619 15:08:04.521116 13557 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:04.523029 13561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:04.523098 13558 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4dcebe747914c8f9fbffbfb3674c8e5. Latest consensus state: current_term: 1 leader_uuid: "e4dcebe747914c8f9fbffbfb3674c8e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dcebe747914c8f9fbffbfb3674c8e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42453 } } }
I20260619 15:08:04.523622 13558 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:04.534014 13561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:04.544273 13561 catalog_manager.cc:1347] Generated new cluster ID: 2946dee54e754916b04cab2a4d292915
I20260619 15:08:04.544416 13561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:04.565636 13561 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:04.566090 13561 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:04.577891 13561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4dcebe747914c8f9fbffbfb3674c8e5: Generated new TSK 0
I20260619 15:08:04.578862 13561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:04.592392 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13479
W20260619 15:08:04.593643 13541 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37235
W20260619 15:08:04.594468 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13479: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:08:04.594642 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 13479
2026-06-19T15:08:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:04 dist-test-slave-zpbw krb5kdc[13578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:04 dist-test-slave-zpbw krb5kdc[13578](info): set up 2 sockets
Jun 19 15:08:04 dist-test-slave-zpbw krb5kdc[13578](info): commencing operation
krb5kdc: starting...
W20260619 15:08:05.715484 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:05 dist-test-slave-zpbw krb5kdc[13578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:08:05.985450 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:06.325807 13591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:06.326122 13591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:06.326216 13591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:06.333731 13591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:06.333860 13591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:06.333909 13591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:06.333969 13591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:06.334013 13591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:06.344209 13591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:06.345544 13591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:06.346951 13591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:06.359510 13597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:06.359474 13596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:06.359645 13599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:06.360790 13591 server_base.cc:1034] running on GCE node
I20260619 15:08:06.361581 13591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:06.363130 13591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:06.364388 13591 hybrid_clock.cc:648] HybridClock initialized: now 1781881686364342 us; error 44 us; skew 500 ppm
I20260619 15:08:06.364794 13591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:06 dist-test-slave-zpbw krb5kdc[13578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:06.368252 13591 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:08:06.369874 13591 webserver.cc:460] Webserver started at http://127.0.0.1:38403/ using document root <none> and password file <none>
I20260619 15:08:06.370544 13591 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:06.370646 13591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:06.371008 13591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:06.374073 13591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "c529a9e4494f4bc893a4982bab7a7810"
format_stamp: "Formatted at 2026-06-19 15:08:06 on dist-test-slave-zpbw"
server_key: "dd0264c8966e30460098cbf78bbb8f62"
server_key_iv: "b9f9e9f2ab155aaf27494c02a0048d6a"
server_key_version: "encryptionkey@0"
I20260619 15:08:06.374930 13591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "c529a9e4494f4bc893a4982bab7a7810"
format_stamp: "Formatted at 2026-06-19 15:08:06 on dist-test-slave-zpbw"
server_key: "dd0264c8966e30460098cbf78bbb8f62"
server_key_iv: "b9f9e9f2ab155aaf27494c02a0048d6a"
server_key_version: "encryptionkey@0"
I20260619 15:08:06.380463 13591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 15:08:06.384357 13606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:06.385856 13591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:08:06.386016 13591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "c529a9e4494f4bc893a4982bab7a7810"
format_stamp: "Formatted at 2026-06-19 15:08:06 on dist-test-slave-zpbw"
server_key: "dd0264c8966e30460098cbf78bbb8f62"
server_key_iv: "b9f9e9f2ab155aaf27494c02a0048d6a"
server_key_version: "encryptionkey@0"
I20260619 15:08:06.386200 13591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:06.400458 13591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:06.405189 13591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:06.405464 13591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:06.425040 13591 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43323
I20260619 15:08:06.425071 13657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43323 every 8 connection(s)
I20260619 15:08:06.426556 13591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:06.430680 13658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:06.432353 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 13591
I20260619 15:08:06.432550 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:06.433023 24989 external_mini_cluster.cc:1428] Setting key f7284ee2bc441a6c2ab2e1dda191a548
I20260619 15:08:06.443986 13658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810: Bootstrap starting.
Jun 19 15:08:06 dist-test-slave-zpbw krb5kdc[13578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 15:08:06.447391 13658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:06.448727 13658 log.cc:824] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:06.452446 13658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810: No bootstrap required, opened a new log
I20260619 15:08:06.457381 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:06.435048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:57089 (local address 127.0.0.1:43323)
0619 15:08:06.435852 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:06.435866 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:06.435914 (+ 48us) server_negotiation.cc:407] Connection header received
0619 15:08:06.437583 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:06.437609 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:06.438062 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:06.438490 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:06.440059 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:06.441375 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:06.442751 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:06.443551 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:06.443774 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:06.448458 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:06.448467 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:06.448481 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:06.448525 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:06.452397 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:06.453293 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:06.453302 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:06.453310 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:06.453523 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:06.454001 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:06.454006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:06.454009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:06.454814 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:06.455188 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:06.455569 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 15:08:06.455955 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":339,"threads_started":1}
I20260619 15:08:06.461728 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:06.462638 13658 raft_consensus.cc:348] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c529a9e4494f4bc893a4982bab7a7810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43323 } }
I20260619 15:08:06.463107 13658 raft_consensus.cc:374] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:06.463233 13658 raft_consensus.cc:729] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c529a9e4494f4bc893a4982bab7a7810, State: Initialized, Role: FOLLOWER
I20260619 15:08:06.463898 13658 consensus_queue.cc:260] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c529a9e4494f4bc893a4982bab7a7810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43323 } }
I20260619 15:08:06.464131 13658 raft_consensus.cc:388] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:06.464296 13658 raft_consensus.cc:482] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:06.464483 13658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:06.467121 13658 raft_consensus.cc:504] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c529a9e4494f4bc893a4982bab7a7810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43323 } }
I20260619 15:08:06.467808 13658 leader_election.cc:302] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c529a9e4494f4bc893a4982bab7a7810; no voters:
I20260619 15:08:06.468562 13658 leader_election.cc:288] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:06.468704 13668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:06.471416 13658 sys_catalog.cc:564] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:06.472252 13668 raft_consensus.cc:686] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [term 1 LEADER]: Becoming Leader. State: Replica: c529a9e4494f4bc893a4982bab7a7810, State: Running, Role: LEADER
I20260619 15:08:06.473043 13668 consensus_queue.cc:237] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c529a9e4494f4bc893a4982bab7a7810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43323 } }
I20260619 15:08:06.476835 13670 sys_catalog.cc:455] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c529a9e4494f4bc893a4982bab7a7810. Latest consensus state: current_term: 1 leader_uuid: "c529a9e4494f4bc893a4982bab7a7810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c529a9e4494f4bc893a4982bab7a7810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43323 } } }
I20260619 15:08:06.476958 13669 sys_catalog.cc:455] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c529a9e4494f4bc893a4982bab7a7810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c529a9e4494f4bc893a4982bab7a7810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43323 } } }
I20260619 15:08:06.477339 13669 sys_catalog.cc:458] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:06.477581 13670 sys_catalog.cc:458] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:06.478360 13678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:06.486541 13678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:06.489549 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:06.466617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:60351 (local address 127.0.0.1:43323)
0619 15:08:06.466945 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:06.466951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:06.467396 (+ 445us) server_negotiation.cc:407] Connection header received
0619 15:08:06.467538 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:06.467549 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:06.467663 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:06.467879 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:06.469646 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:06.470522 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:06.471999 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:06.472673 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:06.472827 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:06.476387 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:06.476396 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:06.476403 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:06.476458 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:06.481532 (+ 5074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:06.483089 (+ 1557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:06.483098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:06.483105 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:06.483337 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:06.484087 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:06.484096 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:06.484101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:06.485235 (+ 1134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:06.488291 (+ 3056us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:06.488610 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 15:08:06.488972 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:08:06.506330 13678 catalog_manager.cc:1347] Generated new cluster ID: 322be152ff384b7388685d341ca58c61
I20260619 15:08:06.506870 13678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:06.540270 13678 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:06.542272 13678 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:06.561110 13678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c529a9e4494f4bc893a4982bab7a7810: Generated new TSK 0
I20260619 15:08:06.562074 13678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:06.575263 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13591
I20260619 15:08:06.576594 13653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:08:06.671907 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 13591
W20260619 15:08:06.689481 13576 connection.cc:535] client connection to 127.0.0.1:43323 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43323): Connection reset by peer (error 104)
2026-06-19T15:08:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2080 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:06 dist-test-slave-zpbw krb5kdc[13690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:06 dist-test-slave-zpbw krb5kdc[13690](info): set up 2 sockets
Jun 19 15:08:06 dist-test-slave-zpbw krb5kdc[13690](info): commencing operation
krb5kdc: starting...
W20260619 15:08:07.803716 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:07 dist-test-slave-zpbw krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:08:08.072191 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:08.409184 13703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:08.409505 13703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:08.409602 13703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:08.417307 13703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:08.417434 13703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:08.417481 13703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:08.417543 13703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:08.417588 13703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:08.427719 13703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:08.429039 13703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:08.430299 13703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:08.442592 13708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:08.442593 13709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:08.442622 13711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:08.443559 13703 server_base.cc:1034] running on GCE node
I20260619 15:08:08.444416 13703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:08.445957 13703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:08.447204 13703 hybrid_clock.cc:648] HybridClock initialized: now 1781881688447162 us; error 63 us; skew 500 ppm
I20260619 15:08:08.447587 13703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:08 dist-test-slave-zpbw krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:08.457178 13703 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:08:08.458947 13703 webserver.cc:460] Webserver started at http://127.0.0.1:46281/ using document root <none> and password file <none>
I20260619 15:08:08.459717 13703 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:08.459882 13703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:08.460217 13703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:08.463482 13703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "33cc769ff8ff4423bcb91c7fb9808064"
format_stamp: "Formatted at 2026-06-19 15:08:08 on dist-test-slave-zpbw"
server_key: "577dd00519be1738204e4ab55965e1a6"
server_key_iv: "2e75e8cec4eed1ee9b29f8c7e7cb4a95"
server_key_version: "encryptionkey@0"
I20260619 15:08:08.464313 13703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "33cc769ff8ff4423bcb91c7fb9808064"
format_stamp: "Formatted at 2026-06-19 15:08:08 on dist-test-slave-zpbw"
server_key: "577dd00519be1738204e4ab55965e1a6"
server_key_iv: "2e75e8cec4eed1ee9b29f8c7e7cb4a95"
server_key_version: "encryptionkey@0"
I20260619 15:08:08.469920 13703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 15:08:08.473721 13718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:08.475224 13703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:08:08.475433 13703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "33cc769ff8ff4423bcb91c7fb9808064"
format_stamp: "Formatted at 2026-06-19 15:08:08 on dist-test-slave-zpbw"
server_key: "577dd00519be1738204e4ab55965e1a6"
server_key_iv: "2e75e8cec4eed1ee9b29f8c7e7cb4a95"
server_key_version: "encryptionkey@0"
I20260619 15:08:08.475656 13703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:08.503165 13703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:08.507516 13703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:08.507845 13703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:08.527526 13703 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36475
I20260619 15:08:08.527549 13769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36475 every 8 connection(s)
I20260619 15:08:08.529121 13703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:08.529484 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 13703
I20260619 15:08:08.529663 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:08.530151 24989 external_mini_cluster.cc:1428] Setting key 7d57fa2f33943d120a64609f734fcb8c
I20260619 15:08:08.534020 13770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:08.544185 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:08.532229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:54141 (local address 127.0.0.1:36475)
0619 15:08:08.533044 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:08.533056 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:08.533104 (+ 48us) server_negotiation.cc:407] Connection header received
0619 15:08:08.535013 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:08.535037 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:08.535445 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:08.535867 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:08.536446 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:08.537793 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:08.539352 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:08.540072 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:08.540309 (+ 237us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:08.541700 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:08.541709 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:08.541725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:08.542304 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:08.542385 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:08.542413 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:08.542602 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:08.542927 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 15:08:08.543128 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":337,"threads_started":1}
I20260619 15:08:08.545892 13770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064: Bootstrap starting.
I20260619 15:08:08.548614 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:08.549731 13770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:08.550949 13770 log.cc:824] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:08.554659 13770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064: No bootstrap required, opened a new log
I20260619 15:08:08.560467 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:08.553445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:60715 (local address 127.0.0.1:36475)
0619 15:08:08.553672 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:08.553689 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:08.554040 (+ 351us) server_negotiation.cc:407] Connection header received
0619 15:08:08.554293 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:08.554298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:08.554360 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:08.554471 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:08.555090 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:08.555881 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:08.557255 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:08.557738 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:08.557892 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:08.559384 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:08.559390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:08.559395 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:08.559419 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:08.559486 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:08.559521 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:08.559706 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:08.559990 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 15:08:08.560063 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 15:08:08.563854 13770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33cc769ff8ff4423bcb91c7fb9808064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } }
I20260619 15:08:08.564306 13770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:08.564417 13770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33cc769ff8ff4423bcb91c7fb9808064, State: Initialized, Role: FOLLOWER
I20260619 15:08:08.565011 13770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33cc769ff8ff4423bcb91c7fb9808064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } }
I20260619 15:08:08.565276 13770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:08.565428 13770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:08.565639 13770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:08.568416 13770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33cc769ff8ff4423bcb91c7fb9808064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } }
I20260619 15:08:08.569111 13770 leader_election.cc:302] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33cc769ff8ff4423bcb91c7fb9808064; no voters:
I20260619 15:08:08.569733 13770 leader_election.cc:288] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:08.569909 13780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:08.570690 13780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [term 1 LEADER]: Becoming Leader. State: Replica: 33cc769ff8ff4423bcb91c7fb9808064, State: Running, Role: LEADER
I20260619 15:08:08.571311 13780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33cc769ff8ff4423bcb91c7fb9808064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } }
I20260619 15:08:08.572073 13770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:08.574394 13782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33cc769ff8ff4423bcb91c7fb9808064. Latest consensus state: current_term: 1 leader_uuid: "33cc769ff8ff4423bcb91c7fb9808064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33cc769ff8ff4423bcb91c7fb9808064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } } }
I20260619 15:08:08.574930 13782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:08.577286 13781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33cc769ff8ff4423bcb91c7fb9808064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33cc769ff8ff4423bcb91c7fb9808064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } } }
I20260619 15:08:08.577796 13781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:08.578329 13788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:08.584272 13788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:08.594388 13788 catalog_manager.cc:1347] Generated new cluster ID: 80422e637d984a95a876f2a4b41e3bf8
I20260619 15:08:08.594527 13788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:08.603513 13788 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:08.604894 13788 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:08.611274 13788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33cc769ff8ff4423bcb91c7fb9808064: Generated new TSK 0
I20260619 15:08:08.612221 13788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:08.627012 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13703
W20260619 15:08:08.628324 13765 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.24:54141
W20260619 15:08:08.629211 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13703: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:08:08.629347 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 13703
W20260619 15:08:08.646133 13688 connection.cc:535] client connection to 127.0.0.1:36475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36475): Connection reset by peer (error 104)
2026-06-19T15:08:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:08 dist-test-slave-zpbw krb5kdc[13801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:08 dist-test-slave-zpbw krb5kdc[13801](info): set up 2 sockets
Jun 19 15:08:08 dist-test-slave-zpbw krb5kdc[13801](info): commencing operation
krb5kdc: starting...
W20260619 15:08:09.767520 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:09 dist-test-slave-zpbw krb5kdc[13801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:08:10.039664 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:10.378898 13814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:10.379317 13814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:10.379491 13814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:10.386953 13814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:10.387151 13814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:10.387257 13814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:10.387377 13814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:10.387462 13814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:10.397722 13814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:10.399216 13814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:10.400602 13814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:10.413347 13819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:10.413260 13822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:10.413369 13820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:10.414019 13814 server_base.cc:1034] running on GCE node
I20260619 15:08:10.415030 13814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:10.416574 13814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:10.417819 13814 hybrid_clock.cc:648] HybridClock initialized: now 1781881690417777 us; error 40 us; skew 500 ppm
I20260619 15:08:10.418198 13814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:10 dist-test-slave-zpbw krb5kdc[13801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:10.421680 13814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:08:10.423305 13814 webserver.cc:460] Webserver started at http://127.0.0.1:46543/ using document root <none> and password file <none>
I20260619 15:08:10.424005 13814 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:10.424110 13814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:10.424381 13814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:10.427567 13814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "1a137a254fe344da8ca7048850cdf15a"
format_stamp: "Formatted at 2026-06-19 15:08:10 on dist-test-slave-zpbw"
server_key: "1d1019b541239552fe6c55703c3355cf"
server_key_iv: "62c2fd335f57481ff565413d4f36c1e6"
server_key_version: "encryptionkey@0"
I20260619 15:08:10.428361 13814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "1a137a254fe344da8ca7048850cdf15a"
format_stamp: "Formatted at 2026-06-19 15:08:10 on dist-test-slave-zpbw"
server_key: "1d1019b541239552fe6c55703c3355cf"
server_key_iv: "62c2fd335f57481ff565413d4f36c1e6"
server_key_version: "encryptionkey@0"
I20260619 15:08:10.434188 13814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 15:08:10.438009 13829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:10.439555 13814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:08:10.439720 13814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "1a137a254fe344da8ca7048850cdf15a"
format_stamp: "Formatted at 2026-06-19 15:08:10 on dist-test-slave-zpbw"
server_key: "1d1019b541239552fe6c55703c3355cf"
server_key_iv: "62c2fd335f57481ff565413d4f36c1e6"
server_key_version: "encryptionkey@0"
I20260619 15:08:10.439904 13814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:10.446273 13814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:10.450982 13814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:10.451256 13814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:10.470716 13814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41739
I20260619 15:08:10.470757 13880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41739 every 8 connection(s)
I20260619 15:08:10.472292 13814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:10.476346 13881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:10.476732 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 13814
I20260619 15:08:10.476915 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:10.477375 24989 external_mini_cluster.cc:1428] Setting key 373a339f6b09bf78d4467f5a16197fe5
I20260619 15:08:10.486362 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:10.479257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:54957 (local address 127.0.0.1:41739)
0619 15:08:10.480162 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:10.480174 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:10.480228 (+ 54us) server_negotiation.cc:407] Connection header received
0619 15:08:10.482557 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:10.482585 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:10.483088 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:10.483478 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:10.484663 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:10.484671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:10.484685 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:10.484702 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:10.484763 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:10.484783 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:10.484958 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:10.485103 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 15:08:10.485345 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":448,"threads_started":1}
I20260619 15:08:10.490641 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a: Bootstrap starting.
I20260619 15:08:10.490753 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:10.494014 13881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:10.495497 13881 log.cc:824] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:10.498555 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:10.495718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:46199 (local address 127.0.0.1:41739)
0619 15:08:10.495901 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:10.495909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:10.496408 (+ 499us) server_negotiation.cc:407] Connection header received
0619 15:08:10.496609 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:10.496613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:10.496663 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:10.496754 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:10.497802 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:10.497807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:10.497812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:10.497834 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:10.497883 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:10.497904 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:10.498034 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:10.498198 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 15:08:10.498271 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 15:08:10.499243 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a: No bootstrap required, opened a new log
I20260619 15:08:10.507035 13881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a137a254fe344da8ca7048850cdf15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41739 } }
I20260619 15:08:10.507500 13881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:10.507629 13881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a137a254fe344da8ca7048850cdf15a, State: Initialized, Role: FOLLOWER
I20260619 15:08:10.508368 13881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a137a254fe344da8ca7048850cdf15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41739 } }
I20260619 15:08:10.508623 13881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:10.508795 13881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:10.508978 13881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:10.511371 13881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a137a254fe344da8ca7048850cdf15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41739 } }
I20260619 15:08:10.512024 13881 leader_election.cc:302] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a137a254fe344da8ca7048850cdf15a; no voters:
I20260619 15:08:10.512619 13881 leader_election.cc:288] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:10.512804 13891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:10.513893 13891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [term 1 LEADER]: Becoming Leader. State: Replica: 1a137a254fe344da8ca7048850cdf15a, State: Running, Role: LEADER
I20260619 15:08:10.514569 13891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a137a254fe344da8ca7048850cdf15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41739 } }
I20260619 15:08:10.515166 13881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:10.517515 13893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a137a254fe344da8ca7048850cdf15a. Latest consensus state: current_term: 1 leader_uuid: "1a137a254fe344da8ca7048850cdf15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a137a254fe344da8ca7048850cdf15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41739 } } }
I20260619 15:08:10.518309 13892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a137a254fe344da8ca7048850cdf15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a137a254fe344da8ca7048850cdf15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41739 } } }
I20260619 15:08:10.518620 13892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:10.518865 13893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:10.519865 13900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:10.525640 13900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:10.535475 13900 catalog_manager.cc:1347] Generated new cluster ID: 88ad363a6dad4d4c85c4f54c4f952db7
I20260619 15:08:10.535614 13900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:10.548630 13900 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:10.549122 13900 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:10.566305 13900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a137a254fe344da8ca7048850cdf15a: Generated new TSK 0
I20260619 15:08:10.567386 13900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:10.582988 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13814
W20260619 15:08:10.584198 13876 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.24:54957
W20260619 15:08:10.584991 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13814: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:08:10.585161 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 13814
2026-06-19T15:08:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:10 dist-test-slave-zpbw krb5kdc[13912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:10 dist-test-slave-zpbw krb5kdc[13912](info): set up 2 sockets
Jun 19 15:08:10 dist-test-slave-zpbw krb5kdc[13912](info): commencing operation
krb5kdc: starting...
W20260619 15:08:11.707554 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:11 dist-test-slave-zpbw krb5kdc[13912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:08:11.981676 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:12.328985 13925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:12.329304 13925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:12.329494 13925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:12.337653 13925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:12.337815 13925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:12.337922 13925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:12.338022 13925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:12.338109 13925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:12.348717 13925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:12.350139 13925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:12.351590 13925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:12.363967 13930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:12.364271 13931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:12.364264 13933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:12.365142 13925 server_base.cc:1034] running on GCE node
I20260619 15:08:12.365989 13925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:12.367585 13925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:12.368830 13925 hybrid_clock.cc:648] HybridClock initialized: now 1781881692368785 us; error 43 us; skew 500 ppm
I20260619 15:08:12.369213 13925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:12 dist-test-slave-zpbw krb5kdc[13912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:12.372653 13925 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:08:12.374228 13925 webserver.cc:460] Webserver started at http://127.0.0.1:35107/ using document root <none> and password file <none>
I20260619 15:08:12.374994 13925 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:12.375104 13925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:12.375366 13925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:12.378402 13925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "997ec33f9ca54e10b0af8cb60878554a"
format_stamp: "Formatted at 2026-06-19 15:08:12 on dist-test-slave-zpbw"
server_key: "e1a6c11929097616bcdea269ae1c9954"
server_key_iv: "38a16a7395e7392c68b824a715d2a037"
server_key_version: "encryptionkey@0"
I20260619 15:08:12.379249 13925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "997ec33f9ca54e10b0af8cb60878554a"
format_stamp: "Formatted at 2026-06-19 15:08:12 on dist-test-slave-zpbw"
server_key: "e1a6c11929097616bcdea269ae1c9954"
server_key_iv: "38a16a7395e7392c68b824a715d2a037"
server_key_version: "encryptionkey@0"
I20260619 15:08:12.384761 13925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 15:08:12.388590 13940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:12.390081 13925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 15:08:12.390245 13925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "997ec33f9ca54e10b0af8cb60878554a"
format_stamp: "Formatted at 2026-06-19 15:08:12 on dist-test-slave-zpbw"
server_key: "e1a6c11929097616bcdea269ae1c9954"
server_key_iv: "38a16a7395e7392c68b824a715d2a037"
server_key_version: "encryptionkey@0"
I20260619 15:08:12.390424 13925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:12.406420 13925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:12.411036 13925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:12.411291 13925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:12.430727 13925 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46179
I20260619 15:08:12.430751 13991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46179 every 8 connection(s)
I20260619 15:08:12.432248 13925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:12.436275 13992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:12.438495 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 13925
I20260619 15:08:12.438719 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:12.439214 24989 external_mini_cluster.cc:1428] Setting key cb8ceb3303235c3c96f488438436b37e
I20260619 15:08:12.448405 13992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a: Bootstrap starting.
I20260619 15:08:12.451834 13992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:12.452917 13992 log.cc:824] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a: Log is configured to *not* fsync() on all Append() calls
Jun 19 15:08:12 dist-test-slave-zpbw krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 15:08:12.456423 13992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a: No bootstrap required, opened a new log
I20260619 15:08:12.462878 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:12.441207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:57513 (local address 127.0.0.1:46179)
0619 15:08:12.442011 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:12.442024 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:12.442065 (+ 41us) server_negotiation.cc:407] Connection header received
0619 15:08:12.443849 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:12.443874 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:12.444298 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:12.444709 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:12.446216 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:12.447708 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:12.449090 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:12.449785 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:12.449992 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:12.454901 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:12.454911 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:12.454929 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:12.454966 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:12.458144 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:12.459154 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:12.459161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:12.459171 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:12.459354 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:12.459862 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:12.459867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:12.459870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:12.460460 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:12.460840 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:12.461358 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 15:08:12.461718 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":336,"threads_started":1}
I20260619 15:08:12.464913 13992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997ec33f9ca54e10b0af8cb60878554a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46179 } }
I20260619 15:08:12.465319 13992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:12.465442 13992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997ec33f9ca54e10b0af8cb60878554a, State: Initialized, Role: FOLLOWER
I20260619 15:08:12.466110 13992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997ec33f9ca54e10b0af8cb60878554a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46179 } }
I20260619 15:08:12.466413 13992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:12.466572 13992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:12.466764 13992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:12.467258 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:12.470127 13992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997ec33f9ca54e10b0af8cb60878554a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46179 } }
I20260619 15:08:12.470865 13992 leader_election.cc:302] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 997ec33f9ca54e10b0af8cb60878554a; no voters:
I20260619 15:08:12.471956 13992 leader_election.cc:288] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:12.472136 14000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:12.473423 14000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [term 1 LEADER]: Becoming Leader. State: Replica: 997ec33f9ca54e10b0af8cb60878554a, State: Running, Role: LEADER
I20260619 15:08:12.474196 14000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997ec33f9ca54e10b0af8cb60878554a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46179 } }
I20260619 15:08:12.474680 13992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:12.477288 14003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 997ec33f9ca54e10b0af8cb60878554a. Latest consensus state: current_term: 1 leader_uuid: "997ec33f9ca54e10b0af8cb60878554a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997ec33f9ca54e10b0af8cb60878554a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46179 } } }
I20260619 15:08:12.477702 14003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:12.478248 14002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "997ec33f9ca54e10b0af8cb60878554a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997ec33f9ca54e10b0af8cb60878554a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46179 } } }
I20260619 15:08:12.478617 14002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:12.483453 14007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:12.489346 14007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:12.499316 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:12.474189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:51261 (local address 127.0.0.1:46179)
0619 15:08:12.474483 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:12.474489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:12.475289 (+ 800us) server_negotiation.cc:407] Connection header received
0619 15:08:12.475489 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:12.475494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:12.475589 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:12.475703 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:12.478354 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:12.480074 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:12.482756 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:12.483533 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:12.483673 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:12.488045 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:12.488061 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:12.488068 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:12.488126 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:12.493186 (+ 5060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:12.494830 (+ 1644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:12.494839 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:12.494846 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:12.495175 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:12.495982 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:12.495990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:12.495996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:12.496540 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:12.496883 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:12.498430 (+ 1547us) server_negotiation.cc:299] Negotiation successful
0619 15:08:12.498748 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260619 15:08:12.505208 14007 catalog_manager.cc:1347] Generated new cluster ID: c256b7ced5d542bf8258664315351a61
I20260619 15:08:12.505364 14007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:12.513984 14007 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:12.515512 14007 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:12.525028 14007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 997ec33f9ca54e10b0af8cb60878554a: Generated new TSK 0
I20260619 15:08:12.525961 14007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:12.533993 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 13925
I20260619 15:08:12.535496 13987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:08:12.627218 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 13925
W20260619 15:08:12.644518 13910 connection.cc:535] client connection to 127.0.0.1:46179 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46179): Connection reset by peer (error 104)
2026-06-19T15:08:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2042 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:12 dist-test-slave-zpbw krb5kdc[14025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:12 dist-test-slave-zpbw krb5kdc[14025](info): set up 2 sockets
Jun 19 15:08:12 dist-test-slave-zpbw krb5kdc[14025](info): commencing operation
krb5kdc: starting...
W20260619 15:08:13.747443 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:13 dist-test-slave-zpbw krb5kdc[14025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:08:14.017593 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:14.356729 14038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:14.357046 14038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:14.357239 14038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:14.364761 14038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:14.364888 14038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:14.365005 14038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:14.365101 14038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:14.365185 14038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:14.375603 14038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:14.377043 14038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:14.378453 14038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:14.391433 14043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:14.391470 14044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:14.391532 14046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:14.392218 14038 server_base.cc:1034] running on GCE node
I20260619 15:08:14.393178 14038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:14.394706 14038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:14.396021 14038 hybrid_clock.cc:648] HybridClock initialized: now 1781881694395982 us; error 35 us; skew 500 ppm
I20260619 15:08:14.396456 14038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:14 dist-test-slave-zpbw krb5kdc[14025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:14.404151 14038 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:08:14.405953 14038 webserver.cc:460] Webserver started at http://127.0.0.1:36665/ using document root <none> and password file <none>
I20260619 15:08:14.406772 14038 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:14.406997 14038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:14.407327 14038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:14.410517 14038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "12e29764215147cf9415f5a504d96faf"
format_stamp: "Formatted at 2026-06-19 15:08:14 on dist-test-slave-zpbw"
server_key: "951c61cf3d12185c6604ebbadda98836"
server_key_iv: "dda2b10d962a003ab65283c9dccc9aba"
server_key_version: "encryptionkey@0"
I20260619 15:08:14.411453 14038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "12e29764215147cf9415f5a504d96faf"
format_stamp: "Formatted at 2026-06-19 15:08:14 on dist-test-slave-zpbw"
server_key: "951c61cf3d12185c6604ebbadda98836"
server_key_iv: "dda2b10d962a003ab65283c9dccc9aba"
server_key_version: "encryptionkey@0"
I20260619 15:08:14.417200 14038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:08:14.421058 14053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:14.422495 14038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:08:14.422716 14038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "12e29764215147cf9415f5a504d96faf"
format_stamp: "Formatted at 2026-06-19 15:08:14 on dist-test-slave-zpbw"
server_key: "951c61cf3d12185c6604ebbadda98836"
server_key_iv: "dda2b10d962a003ab65283c9dccc9aba"
server_key_version: "encryptionkey@0"
I20260619 15:08:14.422999 14038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:14.444697 14038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:14.448911 14038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:14.449234 14038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:14.468851 14038 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45009
I20260619 15:08:14.468881 14104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45009 every 8 connection(s)
I20260619 15:08:14.470335 14038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:14.474360 14105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:14.475324 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 14038
I20260619 15:08:14.475520 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:14.476101 24989 external_mini_cluster.cc:1428] Setting key bf364be5173832764c2ec190f783a21c
I20260619 15:08:14.489471 14105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf: Bootstrap starting.
I20260619 15:08:14.491452 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:14.478161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:57667 (local address 127.0.0.1:45009)
0619 15:08:14.479038 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:14.479052 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:14.479101 (+ 49us) server_negotiation.cc:407] Connection header received
0619 15:08:14.481035 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:14.481064 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:14.481536 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:14.481920 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:14.482605 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:14.484573 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:14.486541 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:14.487361 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:14.487499 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:14.488940 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:14.488955 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:14.488971 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:14.489539 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:14.489686 (+ 147us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:14.489718 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:14.489834 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:14.490149 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 15:08:14.490368 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":397,"threads_started":1}
I20260619 15:08:14.493152 14105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:14.494457 14105 log.cc:824] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:14.495841 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:14.497905 14105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf: No bootstrap required, opened a new log
I20260619 15:08:14.508774 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:14.501435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:49585 (local address 127.0.0.1:45009)
0619 15:08:14.501643 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:14.501650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:14.502222 (+ 572us) server_negotiation.cc:407] Connection header received
0619 15:08:14.502387 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:14.502395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:14.502494 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:14.502622 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:14.503196 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:14.504421 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:14.505787 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:14.506517 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:14.506674 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:14.507886 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:14.507894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:14.507902 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:14.507924 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:14.507988 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:14.508008 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:14.508124 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:14.508299 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 15:08:14.508355 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:08:14.508426 14105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e29764215147cf9415f5a504d96faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45009 } }
I20260619 15:08:14.508996 14105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:14.509122 14105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e29764215147cf9415f5a504d96faf, State: Initialized, Role: FOLLOWER
I20260619 15:08:14.509912 14105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e29764215147cf9415f5a504d96faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45009 } }
I20260619 15:08:14.510223 14105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:14.510406 14105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:14.510605 14105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:14.512902 14105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e29764215147cf9415f5a504d96faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45009 } }
I20260619 15:08:14.513425 14105 leader_election.cc:302] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12e29764215147cf9415f5a504d96faf; no voters:
I20260619 15:08:14.514019 14105 leader_election.cc:288] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:14.514205 14115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:14.515355 14115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [term 1 LEADER]: Becoming Leader. State: Replica: 12e29764215147cf9415f5a504d96faf, State: Running, Role: LEADER
I20260619 15:08:14.516026 14115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e29764215147cf9415f5a504d96faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45009 } }
I20260619 15:08:14.516512 14105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:14.518891 14116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12e29764215147cf9415f5a504d96faf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e29764215147cf9415f5a504d96faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45009 } } }
I20260619 15:08:14.519557 14116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:14.519495 14117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12e29764215147cf9415f5a504d96faf. Latest consensus state: current_term: 1 leader_uuid: "12e29764215147cf9415f5a504d96faf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e29764215147cf9415f5a504d96faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45009 } } }
I20260619 15:08:14.519904 14117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:14.524663 14124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:14.530606 14124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:14.540163 14124 catalog_manager.cc:1347] Generated new cluster ID: 15d5d4b01054408eb7d8ca8ee166a92a
I20260619 15:08:14.540313 14124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:14.550736 14124 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:14.552112 14124 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:14.559397 14124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12e29764215147cf9415f5a504d96faf: Generated new TSK 0
I20260619 15:08:14.560272 14124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:14.570164 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14038
W20260619 15:08:14.572033 14100 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.24:57667
W20260619 15:08:14.573014 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14038: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:08:14.573253 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 14038
W20260619 15:08:14.588934 14023 connection.cc:535] client connection to 127.0.0.1:45009 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45009): Connection reset by peer (error 104)
2026-06-19T15:08:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1944 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:14 dist-test-slave-zpbw krb5kdc[14136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:14 dist-test-slave-zpbw krb5kdc[14136](info): set up 2 sockets
Jun 19 15:08:14 dist-test-slave-zpbw krb5kdc[14136](info): commencing operation
krb5kdc: starting...
W20260619 15:08:15.687821 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:15 dist-test-slave-zpbw krb5kdc[14136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:08:15.958171 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:16.310585 14149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:16.311034 14149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:16.311239 14149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:16.319059 14149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:16.319208 14149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:16.319299 14149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:16.319414 14149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:16.319509 14149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:16.330346 14149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:16.331876 14149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:16.333431 14149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:16.346858 14154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:16.346823 14155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:16.346899 14157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:16.347831 14149 server_base.cc:1034] running on GCE node
I20260619 15:08:16.348768 14149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:16.350314 14149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:16.351558 14149 hybrid_clock.cc:648] HybridClock initialized: now 1781881696351497 us; error 63 us; skew 500 ppm
I20260619 15:08:16.351963 14149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:16 dist-test-slave-zpbw krb5kdc[14136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:16.355548 14149 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:08:16.357254 14149 webserver.cc:460] Webserver started at http://127.0.0.1:41867/ using document root <none> and password file <none>
I20260619 15:08:16.358129 14149 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:16.358287 14149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:16.358614 14149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:16.361939 14149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "4c4e2f48b921464ebb88dbbf12efda78"
format_stamp: "Formatted at 2026-06-19 15:08:16 on dist-test-slave-zpbw"
server_key: "9cad8847a17a76853593592e3327978f"
server_key_iv: "5c85a0f7a6e1c750d918365186d864d6"
server_key_version: "encryptionkey@0"
I20260619 15:08:16.362885 14149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "4c4e2f48b921464ebb88dbbf12efda78"
format_stamp: "Formatted at 2026-06-19 15:08:16 on dist-test-slave-zpbw"
server_key: "9cad8847a17a76853593592e3327978f"
server_key_iv: "5c85a0f7a6e1c750d918365186d864d6"
server_key_version: "encryptionkey@0"
I20260619 15:08:16.368597 14149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 15:08:16.372519 14164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:16.374012 14149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 15:08:16.374256 14149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "4c4e2f48b921464ebb88dbbf12efda78"
format_stamp: "Formatted at 2026-06-19 15:08:16 on dist-test-slave-zpbw"
server_key: "9cad8847a17a76853593592e3327978f"
server_key_iv: "5c85a0f7a6e1c750d918365186d864d6"
server_key_version: "encryptionkey@0"
I20260619 15:08:16.374488 14149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:16.380801 14149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:16.385638 14149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:16.385978 14149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:16.405762 14149 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33827
I20260619 15:08:16.405857 14215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33827 every 8 connection(s)
I20260619 15:08:16.407316 14149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:16.411573 14216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:16.415685 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 14149
I20260619 15:08:16.415913 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:16.416379 24989 external_mini_cluster.cc:1428] Setting key b687a26d8b505caf1fb97304190dbda5
I20260619 15:08:16.424924 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:16.418572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:42909 (local address 127.0.0.1:33827)
0619 15:08:16.419433 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:16.419447 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:16.419491 (+ 44us) server_negotiation.cc:407] Connection header received
0619 15:08:16.421196 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:16.421221 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:16.421617 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:16.422071 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:16.423258 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:16.423265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:16.423281 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:16.423296 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:16.423351 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:16.423369 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:16.423616 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:16.423761 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 15:08:16.423958 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":379,"threads_started":1}
I20260619 15:08:16.425674 14216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78: Bootstrap starting.
I20260619 15:08:16.428893 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:16.429442 14216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:16.430598 14216 log.cc:824] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:16.434031 14216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78: No bootstrap required, opened a new log
I20260619 15:08:16.436506 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:16.433741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:55655 (local address 127.0.0.1:33827)
0619 15:08:16.433961 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:16.433969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:16.434341 (+ 372us) server_negotiation.cc:407] Connection header received
0619 15:08:16.434621 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:16.434625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:16.434687 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:16.434826 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:16.435818 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:16.435823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:16.435826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:16.435840 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:16.435877 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:16.435891 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:16.435978 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:16.436133 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 15:08:16.436187 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 15:08:16.443440 14216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4e2f48b921464ebb88dbbf12efda78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33827 } }
I20260619 15:08:16.443905 14216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:16.444038 14216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4e2f48b921464ebb88dbbf12efda78, State: Initialized, Role: FOLLOWER
I20260619 15:08:16.444810 14216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4e2f48b921464ebb88dbbf12efda78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33827 } }
I20260619 15:08:16.445140 14216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:16.445326 14216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:16.445529 14216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:16.448127 14216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4e2f48b921464ebb88dbbf12efda78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33827 } }
I20260619 15:08:16.448678 14216 leader_election.cc:302] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c4e2f48b921464ebb88dbbf12efda78; no voters:
I20260619 15:08:16.449291 14216 leader_election.cc:288] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:16.449448 14226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:16.450430 14226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [term 1 LEADER]: Becoming Leader. State: Replica: 4c4e2f48b921464ebb88dbbf12efda78, State: Running, Role: LEADER
I20260619 15:08:16.451116 14226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4e2f48b921464ebb88dbbf12efda78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33827 } }
I20260619 15:08:16.451853 14216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:16.454172 14228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c4e2f48b921464ebb88dbbf12efda78. Latest consensus state: current_term: 1 leader_uuid: "4c4e2f48b921464ebb88dbbf12efda78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4e2f48b921464ebb88dbbf12efda78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33827 } } }
I20260619 15:08:16.454602 14228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:16.455005 14227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c4e2f48b921464ebb88dbbf12efda78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4e2f48b921464ebb88dbbf12efda78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33827 } } }
I20260619 15:08:16.455389 14227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:16.464891 14232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:16.471215 14232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:16.480939 14232 catalog_manager.cc:1347] Generated new cluster ID: 9387ee88cbc64a29bc28ad65a0585754
I20260619 15:08:16.481070 14232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:16.505204 14232 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:16.505735 14232 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:16.515254 14232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c4e2f48b921464ebb88dbbf12efda78: Generated new TSK 0
I20260619 15:08:16.516182 14232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:16.526849 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14149
W20260619 15:08:16.528847 14211 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.24:42909
W20260619 15:08:16.529668 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14149: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:08:16.529839 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 14149
2026-06-19T15:08:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1957 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:16 dist-test-slave-zpbw krb5kdc[14247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:16 dist-test-slave-zpbw krb5kdc[14247](info): set up 2 sockets
Jun 19 15:08:16 dist-test-slave-zpbw krb5kdc[14247](info): commencing operation
krb5kdc: starting...
W20260619 15:08:17.663606 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:17 dist-test-slave-zpbw krb5kdc[14247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:08:17.930025 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35261
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:35261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:18.268616 14260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:18.269032 14260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:18.269197 14260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:18.277318 14260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:18.277496 14260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:18.277602 14260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:18.277712 14260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:18.277786 14260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:18.288297 14260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:35261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:18.289717 14260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:18.291218 14260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:18.303550 14265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:18.303512 14266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:18.303467 14268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:18.304569 14260 server_base.cc:1034] running on GCE node
I20260619 15:08:18.305363 14260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:18.306890 14260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:18.308149 14260 hybrid_clock.cc:648] HybridClock initialized: now 1781881698308068 us; error 71 us; skew 500 ppm
I20260619 15:08:18.308547 14260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:18 dist-test-slave-zpbw krb5kdc[14247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881698, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:18.311820 14260 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:08:18.313504 14260 webserver.cc:460] Webserver started at http://127.24.103.126:38521/ using document root <none> and password file <none>
I20260619 15:08:18.314213 14260 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:18.314388 14260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:18.314709 14260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:18.317957 14260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "20c13a6fabb54da6a5824026bf6af867"
format_stamp: "Formatted at 2026-06-19 15:08:18 on dist-test-slave-zpbw"
server_key: "8c9713321600ca8cbf5ccadd4b5c4860"
server_key_iv: "d41f3f3f2715caab7552a31eadad1eef"
server_key_version: "encryptionkey@0"
I20260619 15:08:18.318831 14260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "20c13a6fabb54da6a5824026bf6af867"
format_stamp: "Formatted at 2026-06-19 15:08:18 on dist-test-slave-zpbw"
server_key: "8c9713321600ca8cbf5ccadd4b5c4860"
server_key_iv: "d41f3f3f2715caab7552a31eadad1eef"
server_key_version: "encryptionkey@0"
I20260619 15:08:18.324301 14260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 15:08:18.328009 14275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:18.329433 14260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 15:08:18.329638 14260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "20c13a6fabb54da6a5824026bf6af867"
format_stamp: "Formatted at 2026-06-19 15:08:18 on dist-test-slave-zpbw"
server_key: "8c9713321600ca8cbf5ccadd4b5c4860"
server_key_iv: "d41f3f3f2715caab7552a31eadad1eef"
server_key_version: "encryptionkey@0"
I20260619 15:08:18.329855 14260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:18.348230 14260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:18.352953 14260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:18.353281 14260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:18.372869 14260 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:35261
I20260619 15:08:18.372877 14326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:35261 every 8 connection(s)
I20260619 15:08:18.374429 14260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:18.377043 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 14260
I20260619 15:08:18.377249 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:18.377833 24989 external_mini_cluster.cc:1428] Setting key a6bd39183c2ae0a69576e0f76176624a
I20260619 15:08:18.378931 14327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 15:08:18 dist-test-slave-zpbw krb5kdc[14247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:08:18.394240 14327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867: Bootstrap starting.
I20260619 15:08:18.398015 14327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:18.399467 14327 log.cc:824] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:18.402400 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:18.379764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47207 (local address 127.24.103.126:35261)
0619 15:08:18.380580 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:18.380591 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:18.380633 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:08:18.382223 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:18.382260 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:18.382714 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:18.383160 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:18.385083 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:18.386470 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:18.388273 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:18.388956 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:18.389084 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:18.394410 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:18.394422 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:18.394448 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:18.394495 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:18.397597 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:18.398725 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:18.398731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:18.398737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:18.398973 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:18.399468 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:18.399472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:18.399474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:18.400160 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:18.400550 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:18.401035 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 15:08:18.401387 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":464,"threads_started":1}
I20260619 15:08:18.402765 14327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867: No bootstrap required, opened a new log
I20260619 15:08:18.406917 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:18.411746 14327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c13a6fabb54da6a5824026bf6af867" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35261 } }
I20260619 15:08:18.412324 14327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:18.412449 14327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c13a6fabb54da6a5824026bf6af867, State: Initialized, Role: FOLLOWER
I20260619 15:08:18.413127 14327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c13a6fabb54da6a5824026bf6af867" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35261 } }
I20260619 15:08:18.413383 14327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:18.413520 14327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:18.413709 14327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:18.416669 14327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c13a6fabb54da6a5824026bf6af867" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35261 } }
I20260619 15:08:18.417328 14327 leader_election.cc:302] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20c13a6fabb54da6a5824026bf6af867; no voters:
I20260619 15:08:18.418015 14327 leader_election.cc:288] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:18.418179 14337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:18.423463 14337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [term 1 LEADER]: Becoming Leader. State: Replica: 20c13a6fabb54da6a5824026bf6af867, State: Running, Role: LEADER
I20260619 15:08:18.424243 14327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:18.424249 14337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c13a6fabb54da6a5824026bf6af867" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35261 } }
I20260619 15:08:18.428023 14338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20c13a6fabb54da6a5824026bf6af867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c13a6fabb54da6a5824026bf6af867" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35261 } } }
I20260619 15:08:18.428784 14338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:18.429873 14345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:18.430373 14339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20c13a6fabb54da6a5824026bf6af867. Latest consensus state: current_term: 1 leader_uuid: "20c13a6fabb54da6a5824026bf6af867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c13a6fabb54da6a5824026bf6af867" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35261 } } }
I20260619 15:08:18.430763 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:18.411513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41607 (local address 127.24.103.126:35261)
0619 15:08:18.411698 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:18.411703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:18.412188 (+ 485us) server_negotiation.cc:407] Connection header received
0619 15:08:18.412456 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:18.412464 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:18.412562 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:18.412682 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:18.414548 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:18.415797 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:18.417137 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:18.417830 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:18.417983 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:18.422071 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:18.422081 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:18.422089 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:18.422150 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:18.426686 (+ 4536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:18.427697 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:18.427703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:18.427706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:18.427864 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:18.428367 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:18.428378 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:18.428381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:18.428829 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:18.429148 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:18.430141 (+ 993us) server_negotiation.cc:299] Negotiation successful
0619 15:08:18.430322 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 15:08:18.430824 14339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:18.436762 14345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:18.448906 14345 catalog_manager.cc:1347] Generated new cluster ID: 3228b9c6bd554907a007ac080df792c1
I20260619 15:08:18.449035 14345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:18.471441 14345 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:18.473058 14345 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:18.485401 14345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20c13a6fabb54da6a5824026bf6af867: Generated new TSK 0
I20260619 15:08:18.486450 14345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:18.498987 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14260
I20260619 15:08:18.500221 14322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:08:18.594413 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 14260
W20260619 15:08:18.612432 14245 connection.cc:535] client connection to 127.24.103.126:35261 recv error: Network error: failed to read from TLS socket (remote: 127.24.103.126:35261): Connection reset by peer (error 104)
2026-06-19T15:08:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2065 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:18 dist-test-slave-zpbw krb5kdc[14359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:18 dist-test-slave-zpbw krb5kdc[14359](info): set up 2 sockets
Jun 19 15:08:18 dist-test-slave-zpbw krb5kdc[14359](info): commencing operation
krb5kdc: starting...
W20260619 15:08:19.715701 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:19 dist-test-slave-zpbw krb5kdc[14359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:08:19.984143 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35925
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:35925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:20.329107 14372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:20.329427 14372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:20.329525 14372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:20.337092 14372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:20.337229 14372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:20.337278 14372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:20.337332 14372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:20.337378 14372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:20.347927 14372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:35925
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:20.349354 14372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:20.350746 14372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:20.363850 14378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:20.363888 14377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:20.363921 14380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:20.364501 14372 server_base.cc:1034] running on GCE node
I20260619 15:08:20.365571 14372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:20.367223 14372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:20.368494 14372 hybrid_clock.cc:648] HybridClock initialized: now 1781881700368435 us; error 62 us; skew 500 ppm
I20260619 15:08:20.368924 14372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:20 dist-test-slave-zpbw krb5kdc[14359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881700, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:20.376132 14372 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:08:20.377974 14372 webserver.cc:460] Webserver started at http://127.24.103.126:40411/ using document root <none> and password file <none>
I20260619 15:08:20.378753 14372 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:20.378993 14372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:20.379361 14372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:20.382603 14372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "2c6ac2486e254f368efd7791a598300f"
format_stamp: "Formatted at 2026-06-19 15:08:20 on dist-test-slave-zpbw"
server_key: "b210163bf3fc3bf1a2a0f56b4e35e45f"
server_key_iv: "3852ee3f18cc29d646004f43bd2e5c55"
server_key_version: "encryptionkey@0"
I20260619 15:08:20.383571 14372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "2c6ac2486e254f368efd7791a598300f"
format_stamp: "Formatted at 2026-06-19 15:08:20 on dist-test-slave-zpbw"
server_key: "b210163bf3fc3bf1a2a0f56b4e35e45f"
server_key_iv: "3852ee3f18cc29d646004f43bd2e5c55"
server_key_version: "encryptionkey@0"
I20260619 15:08:20.389253 14372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 15:08:20.393198 14387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:20.394711 14372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:08:20.395015 14372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "2c6ac2486e254f368efd7791a598300f"
format_stamp: "Formatted at 2026-06-19 15:08:20 on dist-test-slave-zpbw"
server_key: "b210163bf3fc3bf1a2a0f56b4e35e45f"
server_key_iv: "3852ee3f18cc29d646004f43bd2e5c55"
server_key_version: "encryptionkey@0"
I20260619 15:08:20.395253 14372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:20.434132 14372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:20.438947 14372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:20.439286 14372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:20.459121 14372 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:35925
I20260619 15:08:20.459141 14438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:35925 every 8 connection(s)
I20260619 15:08:20.460623 14372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:20.461740 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 14372
I20260619 15:08:20.461948 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:20.462487 24989 external_mini_cluster.cc:1428] Setting key 983a3c11d9d611db888adf41641fce75
I20260619 15:08:20.465701 14439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:20.476333 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:20.464517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53925 (local address 127.24.103.126:35925)
0619 15:08:20.465310 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:20.465323 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:20.465362 (+ 39us) server_negotiation.cc:407] Connection header received
0619 15:08:20.467169 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:20.467192 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:20.467604 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:20.468049 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:20.468626 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:20.470107 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:20.471573 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:20.472305 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:20.472513 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:20.473854 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:20.473865 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:20.473881 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:20.474465 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:20.474545 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:20.474572 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:20.474836 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:20.475142 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 15:08:20.475342 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":349,"threads_started":1}
I20260619 15:08:20.477349 14439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f: Bootstrap starting.
I20260619 15:08:20.480473 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:20.480872 14439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:20.481945 14439 log.cc:824] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:20.485258 14439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f: No bootstrap required, opened a new log
I20260619 15:08:20.492689 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:20.485549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54209 (local address 127.24.103.126:35925)
0619 15:08:20.485775 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:20.485783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:20.486148 (+ 365us) server_negotiation.cc:407] Connection header received
0619 15:08:20.486375 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:20.486379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:20.486440 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:20.486517 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:20.487203 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:20.487999 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:20.489945 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:20.490380 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:20.490476 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:20.491857 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:20.491862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:20.491866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:20.491887 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:20.491929 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:20.491945 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:20.492035 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:20.492256 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 15:08:20.492305 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 15:08:20.495668 14439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6ac2486e254f368efd7791a598300f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35925 } }
I20260619 15:08:20.496135 14439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:20.496259 14439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6ac2486e254f368efd7791a598300f, State: Initialized, Role: FOLLOWER
I20260619 15:08:20.496913 14439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6ac2486e254f368efd7791a598300f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35925 } }
I20260619 15:08:20.497176 14439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:20.497331 14439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:20.497529 14439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:20.500417 14439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6ac2486e254f368efd7791a598300f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35925 } }
I20260619 15:08:20.500980 14439 leader_election.cc:302] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c6ac2486e254f368efd7791a598300f; no voters:
I20260619 15:08:20.501582 14439 leader_election.cc:288] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:20.501745 14449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:20.502538 14449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [term 1 LEADER]: Becoming Leader. State: Replica: 2c6ac2486e254f368efd7791a598300f, State: Running, Role: LEADER
I20260619 15:08:20.503127 14449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6ac2486e254f368efd7791a598300f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35925 } }
I20260619 15:08:20.503890 14439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:20.506011 14451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c6ac2486e254f368efd7791a598300f. Latest consensus state: current_term: 1 leader_uuid: "2c6ac2486e254f368efd7791a598300f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6ac2486e254f368efd7791a598300f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35925 } } }
I20260619 15:08:20.506413 14451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:20.507625 14450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c6ac2486e254f368efd7791a598300f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6ac2486e254f368efd7791a598300f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35925 } } }
I20260619 15:08:20.508133 14450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:20.508656 14454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:20.515882 14454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:20.526207 14454 catalog_manager.cc:1347] Generated new cluster ID: 5cbfa13bb775402a9528cee281ecf1f7
I20260619 15:08:20.526379 14454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:20.547958 14454 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:20.549690 14454 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:20.561681 14454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c6ac2486e254f368efd7791a598300f: Generated new TSK 0
I20260619 15:08:20.562629 14454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:20.568459 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14372
W20260619 15:08:20.570057 14434 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53925
W20260619 15:08:20.570940 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14372: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:08:20.571122 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 14372
W20260619 15:08:20.584828 14357 connection.cc:535] client connection to 127.24.103.126:35925 recv error: Network error: failed to read from TLS socket (remote: 127.24.103.126:35925): Connection reset by peer (error 104)
2026-06-19T15:08:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1972 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:20 dist-test-slave-zpbw krb5kdc[14470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:20 dist-test-slave-zpbw krb5kdc[14470](info): set up 2 sockets
Jun 19 15:08:20 dist-test-slave-zpbw krb5kdc[14470](info): commencing operation
krb5kdc: starting...
W20260619 15:08:21.695721 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:21 dist-test-slave-zpbw krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:08:21.970734 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:37765
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:37765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:22.316310 14483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:22.316705 14483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:22.316897 14483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:22.324525 14483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:22.324707 14483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:22.324826 14483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:22.324944 14483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:22.325038 14483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:22.335391 14483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:37765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:37765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:22.336844 14483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:22.338243 14483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:22.351558 14489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:22.351440 14491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:22.351469 14488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:22.352561 14483 server_base.cc:1034] running on GCE node
I20260619 15:08:22.353442 14483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:22.355074 14483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:22.356303 14483 hybrid_clock.cc:648] HybridClock initialized: now 1781881702356236 us; error 57 us; skew 500 ppm
I20260619 15:08:22.356662 14483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:22 dist-test-slave-zpbw krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881702, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:22.360133 14483 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:08:22.361745 14483 webserver.cc:460] Webserver started at http://127.24.103.126:34131/ using document root <none> and password file <none>
I20260619 15:08:22.362476 14483 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:22.362628 14483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:22.363015 14483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:22.366164 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "31905b6cfbe44b94a32436df29467d23"
format_stamp: "Formatted at 2026-06-19 15:08:22 on dist-test-slave-zpbw"
server_key: "3c2a2b1e2db56cc8cc51f943f13cd14d"
server_key_iv: "22b2c68bfb35ab1d2e3833087dd9f329"
server_key_version: "encryptionkey@0"
I20260619 15:08:22.367064 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "31905b6cfbe44b94a32436df29467d23"
format_stamp: "Formatted at 2026-06-19 15:08:22 on dist-test-slave-zpbw"
server_key: "3c2a2b1e2db56cc8cc51f943f13cd14d"
server_key_iv: "22b2c68bfb35ab1d2e3833087dd9f329"
server_key_version: "encryptionkey@0"
I20260619 15:08:22.372933 14483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:08:22.376840 14498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:22.378603 14483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:08:22.378875 14483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "31905b6cfbe44b94a32436df29467d23"
format_stamp: "Formatted at 2026-06-19 15:08:22 on dist-test-slave-zpbw"
server_key: "3c2a2b1e2db56cc8cc51f943f13cd14d"
server_key_iv: "22b2c68bfb35ab1d2e3833087dd9f329"
server_key_version: "encryptionkey@0"
I20260619 15:08:22.379110 14483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:22.385599 14483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:22.390573 14483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:22.390973 14483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:22.411446 14483 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:37765
I20260619 15:08:22.411480 14549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:37765 every 8 connection(s)
I20260619 15:08:22.413026 14483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:22.417210 14550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:22.417780 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 14483
I20260619 15:08:22.417965 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:22.418514 24989 external_mini_cluster.cc:1428] Setting key 16000134079f46e2e67bd369db16fb67
I20260619 15:08:22.427906 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:22.420517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33599 (local address 127.24.103.126:37765)
0619 15:08:22.421447 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:22.421459 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:22.421501 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:08:22.424130 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:22.424161 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:22.424613 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:22.425002 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:22.426212 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:22.426218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:22.426233 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:22.426248 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:22.426311 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:22.426335 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:22.426506 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:22.426646 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 15:08:22.426919 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":485,"threads_started":1}
I20260619 15:08:22.431737 14550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23: Bootstrap starting.
I20260619 15:08:22.432050 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:22.435999 14550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:22.437470 14550 log.cc:824] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:22.440524 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:22.437871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47309 (local address 127.24.103.126:37765)
0619 15:08:22.438075 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:22.438086 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:22.438451 (+ 365us) server_negotiation.cc:407] Connection header received
0619 15:08:22.438608 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:22.438614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:22.438678 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:22.438870 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:22.439826 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:22.439832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:22.439843 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:22.439858 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:22.439897 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:22.439913 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:22.440041 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:22.440175 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 15:08:22.440240 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 15:08:22.441090 14550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23: No bootstrap required, opened a new log
I20260619 15:08:22.449160 14550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31905b6cfbe44b94a32436df29467d23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37765 } }
I20260619 15:08:22.449623 14550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:22.449748 14550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31905b6cfbe44b94a32436df29467d23, State: Initialized, Role: FOLLOWER
I20260619 15:08:22.450544 14550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31905b6cfbe44b94a32436df29467d23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37765 } }
I20260619 15:08:22.450842 14550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:22.450971 14550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:22.451107 14550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:22.453352 14550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31905b6cfbe44b94a32436df29467d23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37765 } }
I20260619 15:08:22.453843 14550 leader_election.cc:302] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31905b6cfbe44b94a32436df29467d23; no voters:
I20260619 15:08:22.454378 14550 leader_election.cc:288] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:22.454551 14560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:22.455358 14560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [term 1 LEADER]: Becoming Leader. State: Replica: 31905b6cfbe44b94a32436df29467d23, State: Running, Role: LEADER
I20260619 15:08:22.456003 14560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31905b6cfbe44b94a32436df29467d23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37765 } }
I20260619 15:08:22.456669 14550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:22.458699 14561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31905b6cfbe44b94a32436df29467d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31905b6cfbe44b94a32436df29467d23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37765 } } }
I20260619 15:08:22.459174 14561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:22.459050 14562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31905b6cfbe44b94a32436df29467d23. Latest consensus state: current_term: 1 leader_uuid: "31905b6cfbe44b94a32436df29467d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31905b6cfbe44b94a32436df29467d23" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37765 } } }
I20260619 15:08:22.459323 14562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:22.460069 14566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:22.468528 14566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:22.479596 14566 catalog_manager.cc:1347] Generated new cluster ID: 128847d5fcc84cc8aacf633ce0325613
I20260619 15:08:22.479782 14566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:22.499771 14566 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:22.500347 14566 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:22.508919 14566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31905b6cfbe44b94a32436df29467d23: Generated new TSK 0
I20260619 15:08:22.509914 14566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:22.517135 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14483
W20260619 15:08:22.518510 14545 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33599
W20260619 15:08:22.519374 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14483: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:08:22.519559 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 14483
2026-06-19T15:08:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:22 dist-test-slave-zpbw krb5kdc[14581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:22 dist-test-slave-zpbw krb5kdc[14581](info): set up 2 sockets
Jun 19 15:08:22 dist-test-slave-zpbw krb5kdc[14581](info): commencing operation
krb5kdc: starting...
W20260619 15:08:23.647559 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:23 dist-test-slave-zpbw krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:08:23.915272 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34529
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:34529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:24.261394 14594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:24.261719 14594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:24.261888 14594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:24.269253 14594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:24.269380 14594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:24.269491 14594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:24.269601 14594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:24.269676 14594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:24.279776 14594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:34529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:24.281236 14594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:24.282572 14594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:24.294914 14599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:24.294965 14600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:24.294935 14602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:24.295776 14594 server_base.cc:1034] running on GCE node
I20260619 15:08:24.296651 14594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:24.298228 14594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:24.299458 14594 hybrid_clock.cc:648] HybridClock initialized: now 1781881704299408 us; error 72 us; skew 500 ppm
I20260619 15:08:24.299829 14594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:24 dist-test-slave-zpbw krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881704, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:24.303040 14594 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:08:24.304554 14594 webserver.cc:460] Webserver started at http://127.24.103.126:34501/ using document root <none> and password file <none>
I20260619 15:08:24.305277 14594 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:24.305428 14594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:24.305752 14594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:24.308923 14594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "7e056230bf4d4d26b3c1936613f11d7d"
format_stamp: "Formatted at 2026-06-19 15:08:24 on dist-test-slave-zpbw"
server_key: "7f4470129d9f2a82458ab5307ac5cc9b"
server_key_iv: "00de51918c957274ebc13c70fc32d543"
server_key_version: "encryptionkey@0"
I20260619 15:08:24.309777 14594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "7e056230bf4d4d26b3c1936613f11d7d"
format_stamp: "Formatted at 2026-06-19 15:08:24 on dist-test-slave-zpbw"
server_key: "7f4470129d9f2a82458ab5307ac5cc9b"
server_key_iv: "00de51918c957274ebc13c70fc32d543"
server_key_version: "encryptionkey@0"
I20260619 15:08:24.315366 14594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:08:24.319047 14609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:24.320487 14594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:08:24.320701 14594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "7e056230bf4d4d26b3c1936613f11d7d"
format_stamp: "Formatted at 2026-06-19 15:08:24 on dist-test-slave-zpbw"
server_key: "7f4470129d9f2a82458ab5307ac5cc9b"
server_key_iv: "00de51918c957274ebc13c70fc32d543"
server_key_version: "encryptionkey@0"
I20260619 15:08:24.320930 14594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:24.352128 14594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:24.356601 14594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:24.356925 14594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:24.376061 14594 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:34529
I20260619 15:08:24.376096 14660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:34529 every 8 connection(s)
I20260619 15:08:24.377513 14594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:24.381834 14661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:24.382529 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 14594
I20260619 15:08:24.382735 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:24.383226 24989 external_mini_cluster.cc:1428] Setting key 556e5a38b7b500a86fa09f1a50efe6b1
I20260619 15:08:24.397166 14661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d: Bootstrap starting.
Jun 19 15:08:24 dist-test-slave-zpbw krb5kdc[14581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:08:24.400732 14661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:24.402204 14661 log.cc:824] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:24.405912 14661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d: No bootstrap required, opened a new log
I20260619 15:08:24.410099 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:24.385180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50583 (local address 127.24.103.126:34529)
0619 15:08:24.386031 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:24.386060 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:24.386113 (+ 53us) server_negotiation.cc:407] Connection header received
0619 15:08:24.388753 (+ 2640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:24.388788 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:24.389406 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:24.389899 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:24.391470 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:24.393566 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:24.394996 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:24.396108 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:24.396328 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:24.400926 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:24.400941 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:24.400963 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:24.401018 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:24.405089 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:24.405887 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:24.405904 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:24.405913 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:24.406145 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:24.406573 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:24.406578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:24.406581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:24.407370 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:24.407677 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:24.408094 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 15:08:24.408592 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":353,"threads_started":1}
I20260619 15:08:24.414397 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:24.416054 14661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e056230bf4d4d26b3c1936613f11d7d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34529 } }
I20260619 15:08:24.416445 14661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:24.416569 14661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e056230bf4d4d26b3c1936613f11d7d, State: Initialized, Role: FOLLOWER
I20260619 15:08:24.417230 14661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e056230bf4d4d26b3c1936613f11d7d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34529 } }
I20260619 15:08:24.417589 14661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:24.417749 14661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:24.417945 14661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:24.421288 14661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e056230bf4d4d26b3c1936613f11d7d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34529 } }
I20260619 15:08:24.422022 14661 leader_election.cc:302] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e056230bf4d4d26b3c1936613f11d7d; no voters:
I20260619 15:08:24.422875 14661 leader_election.cc:288] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:24.422973 14671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:24.425184 14661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:24.426326 14671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [term 1 LEADER]: Becoming Leader. State: Replica: 7e056230bf4d4d26b3c1936613f11d7d, State: Running, Role: LEADER
I20260619 15:08:24.427181 14671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e056230bf4d4d26b3c1936613f11d7d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34529 } }
I20260619 15:08:24.430685 14672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e056230bf4d4d26b3c1936613f11d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e056230bf4d4d26b3c1936613f11d7d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34529 } } }
I20260619 15:08:24.431289 14672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:24.430378 14673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e056230bf4d4d26b3c1936613f11d7d. Latest consensus state: current_term: 1 leader_uuid: "7e056230bf4d4d26b3c1936613f11d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e056230bf4d4d26b3c1936613f11d7d" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34529 } } }
I20260619 15:08:24.431924 14673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:24.434968 14681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:24.440069 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:24.419091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36519 (local address 127.24.103.126:34529)
0619 15:08:24.419335 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:24.419346 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:24.419811 (+ 465us) server_negotiation.cc:407] Connection header received
0619 15:08:24.419982 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:24.419985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:24.420049 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:24.420145 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:24.422166 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:24.423505 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:24.426043 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:24.426955 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:24.427877 (+ 922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:24.430994 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:24.431004 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:24.431013 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:24.431081 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:24.436507 (+ 5426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:24.437625 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:24.437634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:24.437638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:24.437827 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:24.438304 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:24.438309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:24.438315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:24.438785 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:24.439080 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:24.439474 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 15:08:24.439723 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 15:08:24.440905 14681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:24.451282 14681 catalog_manager.cc:1347] Generated new cluster ID: 6005e0ed6e484ee1b52d3d77c806e507
I20260619 15:08:24.451435 14681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:24.465040 14681 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:24.467094 14681 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:24.475067 14681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e056230bf4d4d26b3c1936613f11d7d: Generated new TSK 0
I20260619 15:08:24.476243 14681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:24.489338 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14594
I20260619 15:08:24.490559 14656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:08:24.581655 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 14594
2026-06-19T15:08:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2062 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:24 dist-test-slave-zpbw krb5kdc[14693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:24 dist-test-slave-zpbw krb5kdc[14693](info): set up 2 sockets
Jun 19 15:08:24 dist-test-slave-zpbw krb5kdc[14693](info): commencing operation
krb5kdc: starting...
W20260619 15:08:25.699492 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:25 dist-test-slave-zpbw krb5kdc[14693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:08:25.968255 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39115
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:39115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:26.319422 14706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:26.319841 14706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:26.320039 14706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:26.327701 14706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:26.327916 14706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:26.328049 14706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:26.328173 14706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:26.328271 14706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:26.338800 14706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:39115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:26.340312 14706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:26.341756 14706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:26.355077 14711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:26.355067 14712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:26.355120 14714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:26.355996 14706 server_base.cc:1034] running on GCE node
I20260619 15:08:26.356808 14706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:26.358294 14706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:26.359544 14706 hybrid_clock.cc:648] HybridClock initialized: now 1781881706359486 us; error 57 us; skew 500 ppm
I20260619 15:08:26.359928 14706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:26 dist-test-slave-zpbw krb5kdc[14693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881706, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:26.368254 14706 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:08:26.369951 14706 webserver.cc:460] Webserver started at http://127.24.103.126:38381/ using document root <none> and password file <none>
I20260619 15:08:26.370702 14706 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:26.370877 14706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:26.371212 14706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:26.374382 14706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "fd13dae26c2f42aa9e3e879c6760f728"
format_stamp: "Formatted at 2026-06-19 15:08:26 on dist-test-slave-zpbw"
server_key: "f2c85c736c69bec7830e04e11859ab6f"
server_key_iv: "fb5fe9749cad6805aabcf2ad2535fa16"
server_key_version: "encryptionkey@0"
I20260619 15:08:26.375254 14706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "fd13dae26c2f42aa9e3e879c6760f728"
format_stamp: "Formatted at 2026-06-19 15:08:26 on dist-test-slave-zpbw"
server_key: "f2c85c736c69bec7830e04e11859ab6f"
server_key_iv: "fb5fe9749cad6805aabcf2ad2535fa16"
server_key_version: "encryptionkey@0"
I20260619 15:08:26.380735 14706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 15:08:26.384411 14721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:26.385924 14706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 15:08:26.386117 14706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "fd13dae26c2f42aa9e3e879c6760f728"
format_stamp: "Formatted at 2026-06-19 15:08:26 on dist-test-slave-zpbw"
server_key: "f2c85c736c69bec7830e04e11859ab6f"
server_key_iv: "fb5fe9749cad6805aabcf2ad2535fa16"
server_key_version: "encryptionkey@0"
I20260619 15:08:26.386337 14706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:26.412070 14706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:26.416527 14706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:26.416849 14706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:26.436213 14706 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:39115
I20260619 15:08:26.436223 14772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:39115 every 8 connection(s)
I20260619 15:08:26.437686 14706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:26.441537 14773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:26.446209 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 14706
I20260619 15:08:26.446399 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:26.447021 24989 external_mini_cluster.cc:1428] Setting key d8e27659464394eda9242ecb32738145
I20260619 15:08:26.453485 14773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728: Bootstrap starting.
I20260619 15:08:26.457376 14773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:26.458859 14773 log.cc:824] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:26.461714 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:26.448928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42175 (local address 127.24.103.126:39115)
0619 15:08:26.449745 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:26.449757 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:26.449800 (+ 43us) server_negotiation.cc:407] Connection header received
0619 15:08:26.451858 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:26.451884 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:26.452328 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:26.452668 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:26.453328 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:26.454761 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:26.456799 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:26.457603 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:26.457753 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:26.459104 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:26.459113 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:26.459127 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:26.459903 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:26.459998 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:26.460034 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:26.460192 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:26.460374 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 15:08:26.460668 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":397,"threads_started":1}
I20260619 15:08:26.462357 14773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728: No bootstrap required, opened a new log
I20260619 15:08:26.465754 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:26.470844 14773 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd13dae26c2f42aa9e3e879c6760f728" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39115 } }
I20260619 15:08:26.471318 14773 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:26.471446 14773 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd13dae26c2f42aa9e3e879c6760f728, State: Initialized, Role: FOLLOWER
I20260619 15:08:26.472234 14773 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd13dae26c2f42aa9e3e879c6760f728" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39115 } }
I20260619 15:08:26.472497 14773 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:26.472666 14773 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:26.472846 14773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:26.476193 14773 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd13dae26c2f42aa9e3e879c6760f728" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39115 } }
I20260619 15:08:26.476856 14773 leader_election.cc:302] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd13dae26c2f42aa9e3e879c6760f728; no voters:
I20260619 15:08:26.477596 14773 leader_election.cc:288] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:26.477741 14783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:26.477847 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:26.470265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60951 (local address 127.24.103.126:39115)
0619 15:08:26.470436 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:26.470441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:26.471040 (+ 599us) server_negotiation.cc:407] Connection header received
0619 15:08:26.471275 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:26.471284 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:26.471373 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:26.471492 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:26.472162 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:26.473129 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:26.475117 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:26.475572 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:26.475673 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:26.476959 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:26.476963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:26.476967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:26.476987 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:26.477055 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:26.477077 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:26.477162 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:26.477348 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 15:08:26.477472 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 15:08:26.478991 14783 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [term 1 LEADER]: Becoming Leader. State: Replica: fd13dae26c2f42aa9e3e879c6760f728, State: Running, Role: LEADER
I20260619 15:08:26.479691 14783 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd13dae26c2f42aa9e3e879c6760f728" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39115 } }
I20260619 15:08:26.480134 14773 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:26.483119 14785 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd13dae26c2f42aa9e3e879c6760f728. Latest consensus state: current_term: 1 leader_uuid: "fd13dae26c2f42aa9e3e879c6760f728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd13dae26c2f42aa9e3e879c6760f728" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39115 } } }
I20260619 15:08:26.483073 14784 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd13dae26c2f42aa9e3e879c6760f728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd13dae26c2f42aa9e3e879c6760f728" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39115 } } }
I20260619 15:08:26.483531 14785 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:26.483569 14784 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:26.485538 14790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:26.491009 14790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:26.499902 14790 catalog_manager.cc:1347] Generated new cluster ID: 4fe421b65cdf4d17b00aae47651b879b
I20260619 15:08:26.500034 14790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:26.516469 14790 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:26.518168 14790 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:26.525061 14790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd13dae26c2f42aa9e3e879c6760f728: Generated new TSK 0
I20260619 15:08:26.526258 14790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:26.543272 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14706
W20260619 15:08:26.544984 14768 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42175
W20260619 15:08:26.545742 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14706: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:08:26.545908 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 14706
2026-06-19T15:08:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1963 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:26 dist-test-slave-zpbw krb5kdc[14804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:26 dist-test-slave-zpbw krb5kdc[14804](info): set up 2 sockets
Jun 19 15:08:26 dist-test-slave-zpbw krb5kdc[14804](info): commencing operation
krb5kdc: starting...
W20260619 15:08:27.663897 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:27 dist-test-slave-zpbw krb5kdc[14804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:08:27.931872 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:46711
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:46711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:28.271306 14817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:28.271605 14817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:28.271826 14817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:28.279241 14817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:28.279366 14817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:28.279453 14817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:28.279554 14817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:28.279623 14817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:28.289917 14817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:46711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:46711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:28.291326 14817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:28.292680 14817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:28.304740 14822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:28.304749 14823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:28.304795 14825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:28.305637 14817 server_base.cc:1034] running on GCE node
I20260619 15:08:28.306373 14817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:28.307890 14817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:28.309098 14817 hybrid_clock.cc:648] HybridClock initialized: now 1781881708309056 us; error 31 us; skew 500 ppm
I20260619 15:08:28.309424 14817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:28 dist-test-slave-zpbw krb5kdc[14804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881708, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:28.312525 14817 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:08:28.314085 14817 webserver.cc:460] Webserver started at http://127.24.103.126:39937/ using document root <none> and password file <none>
I20260619 15:08:28.314767 14817 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:28.314937 14817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:28.315243 14817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:28.318276 14817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "56d2cacc03e44a9485945ce9df1f80c3"
format_stamp: "Formatted at 2026-06-19 15:08:28 on dist-test-slave-zpbw"
server_key: "19f200c2134e71c62969eb07b6ff56fd"
server_key_iv: "e75f2940914c235b55e88ade6609f53b"
server_key_version: "encryptionkey@0"
I20260619 15:08:28.319113 14817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "56d2cacc03e44a9485945ce9df1f80c3"
format_stamp: "Formatted at 2026-06-19 15:08:28 on dist-test-slave-zpbw"
server_key: "19f200c2134e71c62969eb07b6ff56fd"
server_key_iv: "e75f2940914c235b55e88ade6609f53b"
server_key_version: "encryptionkey@0"
I20260619 15:08:28.324425 14817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 15:08:28.327975 14832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:28.329327 14817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:08:28.329473 14817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "56d2cacc03e44a9485945ce9df1f80c3"
format_stamp: "Formatted at 2026-06-19 15:08:28 on dist-test-slave-zpbw"
server_key: "19f200c2134e71c62969eb07b6ff56fd"
server_key_iv: "e75f2940914c235b55e88ade6609f53b"
server_key_version: "encryptionkey@0"
I20260619 15:08:28.329648 14817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:28.335642 14817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:28.340010 14817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:28.340276 14817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:28.359383 14817 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:46711
I20260619 15:08:28.359417 14883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:46711 every 8 connection(s)
I20260619 15:08:28.360893 14817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:28.364750 14884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:28.368697 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 14817
I20260619 15:08:28.368885 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:28.369400 24989 external_mini_cluster.cc:1428] Setting key 33d82ae839645bec0343c12d9cd57cd7
I20260619 15:08:28.377295 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:28.371342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45099 (local address 127.24.103.126:46711)
0619 15:08:28.372120 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:28.372131 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:28.372170 (+ 39us) server_negotiation.cc:407] Connection header received
0619 15:08:28.373790 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:28.373813 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:28.374187 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:28.374586 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:28.375717 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:28.375723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:28.375736 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:28.375749 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:28.375803 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:28.375820 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:28.376056 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:28.376209 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 15:08:28.376398 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":348,"threads_started":1}
I20260619 15:08:28.377610 14884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3: Bootstrap starting.
I20260619 15:08:28.381111 14884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:28.381275 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:28.382282 14884 log.cc:824] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:28.385174 14884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3: No bootstrap required, opened a new log
I20260619 15:08:28.389189 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:28.386293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41643 (local address 127.24.103.126:46711)
0619 15:08:28.386487 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:28.386494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:28.387097 (+ 603us) server_negotiation.cc:407] Connection header received
0619 15:08:28.387271 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:28.387275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:28.387331 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:28.387425 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:28.388435 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:28.388439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:28.388442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:08:28.388462 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:28.388504 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:08:28.388524 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:28.388620 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:28.388763 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 15:08:28.388816 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 15:08:28.394513 14884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cacc03e44a9485945ce9df1f80c3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46711 } }
I20260619 15:08:28.395017 14884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:08:28.395126 14884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d2cacc03e44a9485945ce9df1f80c3, State: Initialized, Role: FOLLOWER
I20260619 15:08:28.395789 14884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cacc03e44a9485945ce9df1f80c3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46711 } }
I20260619 15:08:28.396034 14884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:08:28.396201 14884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:08:28.396381 14884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:28.398654 14884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cacc03e44a9485945ce9df1f80c3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46711 } }
I20260619 15:08:28.399250 14884 leader_election.cc:302] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d2cacc03e44a9485945ce9df1f80c3; no voters:
I20260619 15:08:28.399901 14884 leader_election.cc:288] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:08:28.400053 14894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:28.401217 14894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [term 1 LEADER]: Becoming Leader. State: Replica: 56d2cacc03e44a9485945ce9df1f80c3, State: Running, Role: LEADER
I20260619 15:08:28.401944 14894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cacc03e44a9485945ce9df1f80c3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46711 } }
I20260619 15:08:28.402288 14884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:28.405055 14895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56d2cacc03e44a9485945ce9df1f80c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cacc03e44a9485945ce9df1f80c3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46711 } } }
I20260619 15:08:28.405085 14896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d2cacc03e44a9485945ce9df1f80c3. Latest consensus state: current_term: 1 leader_uuid: "56d2cacc03e44a9485945ce9df1f80c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cacc03e44a9485945ce9df1f80c3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 46711 } } }
I20260619 15:08:28.405566 14896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:28.405566 14895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:28.409994 14903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:08:28.415370 14903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:28.425034 14903 catalog_manager.cc:1347] Generated new cluster ID: f867b9ced7224e2bbc74316e43c5df9e
I20260619 15:08:28.425199 14903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:28.437079 14903 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:28.437527 14903 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:28.450044 14903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56d2cacc03e44a9485945ce9df1f80c3: Generated new TSK 0
I20260619 15:08:28.451083 14903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:28.471271 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14817
W20260619 15:08:28.472466 14879 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45099
W20260619 15:08:28.473255 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14817: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 15:08:28.473419 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 14817
2026-06-19T15:08:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1927 ms)
[----------] 18 tests from AuthTokenIssuingTest (35892 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:28 dist-test-slave-zpbw krb5kdc[14915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:28 dist-test-slave-zpbw krb5kdc[14915](info): set up 2 sockets
Jun 19 15:08:28 dist-test-slave-zpbw krb5kdc[14915](info): commencing operation
krb5kdc: starting...
W20260619 15:08:29.587652 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:29 dist-test-slave-zpbw krb5kdc[14915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:08:29.854465 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43431
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:43431,127.24.103.125:38449,127.24.103.124:39841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:30.196504 14928 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 15:08:30.196898 14928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:30.197026 14928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:30.197170 14928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:30.204620 14928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:30.204725 14928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:30.204820 14928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:30.204900 14928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:30.205014 14928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:30.215387 14928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:43431,127.24.103.125:38449,127.24.103.124:39841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:30.216763 14928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:30.218092 14928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:30.230386 14933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:30.230387 14934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:30.230436 14936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:30.231343 14928 server_base.cc:1034] running on GCE node
I20260619 15:08:30.232125 14928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:30.233498 14928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:30.234845 14928 hybrid_clock.cc:648] HybridClock initialized: now 1781881710234694 us; error 39 us; skew 500 ppm
I20260619 15:08:30.235229 14928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:30 dist-test-slave-zpbw krb5kdc[14915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881710, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:30.238423 14928 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:08:30.240070 14928 webserver.cc:460] Webserver started at http://127.24.103.126:46883/ using document root <none> and password file <none>
I20260619 15:08:30.240810 14928 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:30.240952 14928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:30.241290 14928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:30.244503 14928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "8e08c779082d4d4e91a5f43d12536428"
format_stamp: "Formatted at 2026-06-19 15:08:30 on dist-test-slave-zpbw"
server_key: "f79a9ebae34b7c21ce40cf8518f9cf31"
server_key_iv: "60deaf4b3c89bfb8bece27f860f0ee05"
server_key_version: "encryptionkey@0"
I20260619 15:08:30.245366 14928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "8e08c779082d4d4e91a5f43d12536428"
format_stamp: "Formatted at 2026-06-19 15:08:30 on dist-test-slave-zpbw"
server_key: "f79a9ebae34b7c21ce40cf8518f9cf31"
server_key_iv: "60deaf4b3c89bfb8bece27f860f0ee05"
server_key_version: "encryptionkey@0"
I20260619 15:08:30.251024 14928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:08:30.254912 14943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:30.256469 14928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:08:30.256700 14928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "8e08c779082d4d4e91a5f43d12536428"
format_stamp: "Formatted at 2026-06-19 15:08:30 on dist-test-slave-zpbw"
server_key: "f79a9ebae34b7c21ce40cf8518f9cf31"
server_key_iv: "60deaf4b3c89bfb8bece27f860f0ee05"
server_key_version: "encryptionkey@0"
I20260619 15:08:30.256958 14928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:30.279914 14928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:30.284185 14928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:30.284489 14928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:30.303579 14928 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:43431
I20260619 15:08:30.303597 14994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:43431 every 8 connection(s)
I20260619 15:08:30.304980 14928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:30.308813 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:30.311525 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 14928
I20260619 15:08:30.311748 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:30.312528 24989 external_mini_cluster.cc:1428] Setting key ddb0b490c961560be46ae5af32d3e51b
I20260619 15:08:30.318444 14995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:08:30 dist-test-slave-zpbw krb5kdc[14915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881710, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.125@KRBTEST.COM" created.
I20260619 15:08:30.344833 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.322881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.24.103.126:43431)
0619 15:08:30.324896 (+ 2015us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:30.324905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:30.324957 (+ 52us) server_negotiation.cc:407] Connection header received
0619 15:08:30.326249 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:30.326261 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:30.326574 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:30.326840 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:30.328052 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:30.329972 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:30.331449 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:30.332436 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:30.332646 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:30.336195 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:30.336208 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:30.336230 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:30.336274 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:30.340424 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:30.341173 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:30.341184 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:30.341189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:30.341381 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:30.341815 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:30.341823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:30.341828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:30.342401 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:30.342605 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:30.343304 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 15:08:30.343571 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":1775,"spinlock_wait_cycles":45440,"thread_start_us":1523,"threads_started":1}
I20260619 15:08:30.344880 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.322881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:43431 (local address 127.0.0.1:39432)
0619 15:08:30.323868 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.323924 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:30.325844 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:30.326900 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:30.326906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:30.327487 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:30.327821 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:30.327838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:30.330128 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:30.330132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:30.331302 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:30.331318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:30.332659 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:30.332663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:30.332874 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:30.333579 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:30.333607 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:30.335921 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:30.340583 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:30.340595 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:30.340605 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:30.341007 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:30.341493 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:30.341496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:30.341498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:30.341674 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:30.342651 (+ 977us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:30.342657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:30.342937 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 15:08:30.343163 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 15:08:30.343710 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":128,"thread_start_us":393,"threads_started":1}
I20260619 15:08:30.347869 14995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:08:30.349119 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.348599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50500)
0619 15:08:30.348807 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.348958 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 15:08:30.349433 14945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
I20260619 15:08:30.351121 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.350656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50514)
0619 15:08:30.350849 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.350943 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 15:08:30.351331 14995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:38449: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.125.keytab.
Entry for principal kudu/127.24.103.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.125.keytab.
I20260619 15:08:30.374221 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:38449
--webserver_interface=127.24.103.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.125.keytab
--principal=kudu/127.24.103.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:43431,127.24.103.125:38449,127.24.103.124:39841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:08:30.400701 14995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } attempt: 1
I20260619 15:08:30.401983 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.401479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50518)
0619 15:08:30.401678 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.401805 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 15:08:30.403018 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.402635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50532)
0619 15:08:30.402832 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.402899 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 15:08:30.403223 14995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:38449: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
I20260619 15:08:30.471536 14995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } attempt: 2
I20260619 15:08:30.472836 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.472292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50540)
0619 15:08:30.472580 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.472673 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 15:08:30.474066 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.473680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50548)
0619 15:08:30.473829 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.473902 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 15:08:30.474361 14995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:38449: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
I20260619 15:08:30.565615 14995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } attempt: 3
I20260619 15:08:30.566893 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.566389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50554)
0619 15:08:30.566599 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.566700 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 15:08:30.567998 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.567684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50566)
0619 15:08:30.567808 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.567873 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 15:08:30.568264 14995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:38449: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
I20260619 15:08:30.711522 14995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } attempt: 4
I20260619 15:08:30.712791 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.712241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50570)
0619 15:08:30.712563 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.712652 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260619 15:08:30.713860 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.713534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50576)
0619 15:08:30.713666 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.713741 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 15:08:30.714124 14995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:38449: Network error: Client connection negotiation failed: client connection to 127.24.103.125:38449: connect: Connection refused (error 111)
W20260619 15:08:30.723067 15002 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 15:08:30.723455 15002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:30.723567 15002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:30.723693 15002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:30.731228 15002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:30.731323 15002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:30.731449 15002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:30.731539 15002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:30.731652 15002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:30.741999 15002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:43431,127.24.103.125:38449,127.24.103.124:39841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.125.keytab
--principal=kudu/127.24.103.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:38449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.103.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:30.743436 15002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:30.744772 15002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:30.756484 15008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:30.756613 15010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:30.756624 15007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:30.757220 15002 server_base.cc:1034] running on GCE node
I20260619 15:08:30.758014 15002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:30.759693 15002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:30.760967 15002 hybrid_clock.cc:648] HybridClock initialized: now 1781881710760884 us; error 54 us; skew 500 ppm
I20260619 15:08:30.761340 15002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:30 dist-test-slave-zpbw krb5kdc[14915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881710, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:30.764631 15002 init.cc:377] Logged in from keytab as kudu/127.24.103.125@KRBTEST.COM (short username kudu)
I20260619 15:08:30.766168 15002 webserver.cc:460] Webserver started at http://127.24.103.125:35499/ using document root <none> and password file <none>
I20260619 15:08:30.766957 15002 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:30.767094 15002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:30.767426 15002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:30.770464 15002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/data/instance:
uuid: "35303389fc1d4346bd10bdbe921e7e44"
format_stamp: "Formatted at 2026-06-19 15:08:30 on dist-test-slave-zpbw"
server_key: "5bdf42403ef511d5b36c4cc25856e529"
server_key_iv: "1449e93f47372f2dbfa98d918f6fca6b"
server_key_version: "encryptionkey@0"
I20260619 15:08:30.771342 15002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/wal/instance:
uuid: "35303389fc1d4346bd10bdbe921e7e44"
format_stamp: "Formatted at 2026-06-19 15:08:30 on dist-test-slave-zpbw"
server_key: "5bdf42403ef511d5b36c4cc25856e529"
server_key_iv: "1449e93f47372f2dbfa98d918f6fca6b"
server_key_version: "encryptionkey@0"
I20260619 15:08:30.776407 15002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 15:08:30.779924 15017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:30.781284 15002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:08:30.781486 15002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/wal
uuid: "35303389fc1d4346bd10bdbe921e7e44"
format_stamp: "Formatted at 2026-06-19 15:08:30 on dist-test-slave-zpbw"
server_key: "5bdf42403ef511d5b36c4cc25856e529"
server_key_iv: "1449e93f47372f2dbfa98d918f6fca6b"
server_key_version: "encryptionkey@0"
I20260619 15:08:30.781725 15002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:30.798090 15002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:30.802309 15002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:30.802597 15002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:30.822671 15002 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.125:38449
I20260619 15:08:30.822710 15068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.125:38449 every 8 connection(s)
I20260619 15:08:30.824183 15002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
I20260619 15:08:30.828126 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:30.830148 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 15002
I20260619 15:08:30.830377 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-1/wal/instance
I20260619 15:08:30.830852 24989 external_mini_cluster.cc:1428] Setting key 71f5686a14df3bff994666e8727ccf03
I20260619 15:08:30.835052 15069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:08:30 dist-test-slave-zpbw krb5kdc[14915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881710, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.124@KRBTEST.COM" created.
I20260619 15:08:30.857916 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.837808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.24.103.126:43431)
0619 15:08:30.837988 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:30.837993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:30.839485 (+ 1492us) server_negotiation.cc:407] Connection header received
0619 15:08:30.841499 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:30.841507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:30.841601 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:30.841722 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:30.843287 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:30.844444 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:30.845940 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:30.846740 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:30.846924 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:30.850826 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:30.850834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:30.850841 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:30.850884 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:30.854621 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:30.855568 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:30.855573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:30.855576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:30.855705 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:30.856174 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:30.856179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:30.856181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:30.856514 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:30.856647 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:30.857356 (+ 709us) server_negotiation.cc:299] Negotiation successful
0619 15:08:30.857541 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 15:08:30.859200 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.838444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:43431 (local address 127.0.0.1:39438)
0619 15:08:30.839291 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.839346 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:30.841222 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:30.841915 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:30.841923 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:30.842433 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:30.843101 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:30.843120 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:30.844599 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:30.844602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:30.845789 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:30.845805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:30.847012 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:30.847017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:30.847229 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:30.847951 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:30.847974 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:30.850520 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:30.854856 (+ 4336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:30.854864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:30.854880 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:30.855443 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:30.855808 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:30.855811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:30.855814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:30.855982 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:30.856643 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:30.856650 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:30.857004 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 15:08:30.857239 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 15:08:30.857651 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":385,"threads_started":1}
I20260619 15:08:30.861047 15069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:08:30 dist-test-slave-zpbw krb5kdc[14915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881710, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260619 15:08:30.889284 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.862058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50584)
0619 15:08:30.862261 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.862282 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:30.862401 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:30.872125 (+ 9724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:30.872129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:30.872556 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:30.872700 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:30.872709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:30.874982 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:30.874987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:30.875893 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:30.875907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:30.877272 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:30.877277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:30.877438 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:30.878156 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:30.878174 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:30.880904 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:30.885779 (+ 4875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:30.885788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:30.885794 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:30.886514 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:30.887000 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:30.887005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:30.887008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:30.887197 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:30.888068 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:30.888076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:30.888266 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 15:08:30.888452 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 15:08:30.888858 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 15:08:30.889443 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.862150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.24.103.125:38449)
0619 15:08:30.871434 (+ 9284us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:30.871446 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:30.871488 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:08:30.871568 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:30.871579 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:30.871886 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:30.872093 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:30.872895 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:30.874666 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:30.876067 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:30.877084 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:30.877232 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:30.881250 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:30.881270 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:30.881294 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:30.881339 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:30.885617 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:30.886676 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:30.886684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:30.886688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:30.886871 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:30.887314 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:30.887337 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:30.887343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:30.887848 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:30.888074 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:30.888673 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 15:08:30.889011 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":9016,"thread_start_us":8862,"threads_started":1}
I20260619 15:08:30.892402 15069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:08:30.893661 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.893126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:59968)
0619 15:08:30.893316 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.893446 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.124.keytab.
Entry for principal kudu/127.24.103.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.124.keytab.
W20260619 15:08:30.893936 15020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
I20260619 15:08:30.895243 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:39841
--webserver_interface=127.24.103.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.124.keytab
--principal=kudu/127.24.103.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:43431,127.24.103.125:38449,127.24.103.124:39841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:08:30.895687 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.895249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:59972)
0619 15:08:30.895469 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.895569 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 15:08:30.895958 15069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:39841: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
I20260619 15:08:30.945278 15069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } attempt: 1
I20260619 15:08:30.946673 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.946003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:59988)
0619 15:08:30.946278 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.946437 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 15:08:30.947959 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:30.947426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:59996)
0619 15:08:30.947670 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:30.947792 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 15:08:30.948204 15069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:39841: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
I20260619 15:08:31.013478 14995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } attempt: 5
I20260619 15:08:31.016448 15069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } attempt: 2
I20260619 15:08:31.017531 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.017034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60010)
0619 15:08:31.017238 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.017345 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 15:08:31.018689 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.018282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60026)
0619 15:08:31.018438 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.018533 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 15:08:31.019032 15069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:39841: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Jun 19 15:08:31 dist-test-slave-zpbw krb5kdc[14915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881710, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260619 15:08:31.027983 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.014317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50596)
0619 15:08:31.014549 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.014572 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:31.014740 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:31.015165 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:31.015170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:31.015600 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:31.015793 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:31.015803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.016962 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.016969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:31.018246 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:31.018263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.019254 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.019260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:31.019490 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.020135 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:31.020154 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:31.022187 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:31.025336 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:31.025341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:31.025346 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:31.025825 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:31.026227 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:31.026232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:31.026236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:31.026385 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:31.026989 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:31.026994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:31.027164 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 15:08:31.027316 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 15:08:31.027639 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 15:08:31.027981 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.014437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.24.103.125:38449)
0619 15:08:31.014608 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.014615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.014649 (+ 34us) server_negotiation.cc:407] Connection header received
0619 15:08:31.014956 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.014960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.015024 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.015141 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.015957 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.016825 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.018422 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.019089 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.019236 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.022385 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:31.022390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:31.022393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:31.022425 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:31.025175 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.025953 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.025959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.025964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.026094 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.026495 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.026502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.026505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.026864 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:31.027016 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:31.027426 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 15:08:31.027639 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 15:08:31.028970 14995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:08:31.030196 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.029790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60042)
0619 15:08:31.029973 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.030057 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 15:08:31.031240 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.030891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60056)
0619 15:08:31.031029 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.031129 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 15:08:31.031461 14995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:39841: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
I20260619 15:08:31.082837 14995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } attempt: 1
I20260619 15:08:31.084059 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.083598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60058)
0619 15:08:31.083791 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.083886 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 15:08:31.085129 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.084798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60074)
0619 15:08:31.084944 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.085024 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 15:08:31.085337 14995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:39841: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
I20260619 15:08:31.110275 15069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } attempt: 3
I20260619 15:08:31.111522 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.111049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60080)
0619 15:08:31.111254 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.111360 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 15:08:31.112711 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.112282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60088)
0619 15:08:31.112520 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.112593 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260619 15:08:31.112931 15069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:39841: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
I20260619 15:08:31.153569 14995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } attempt: 2
I20260619 15:08:31.154850 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.154288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60096)
0619 15:08:31.154515 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.154598 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 15:08:31.155884 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.155559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60108)
0619 15:08:31.155722 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.155785 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 15:08:31.156152 14995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:39841: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
W20260619 15:08:31.241776 15076 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 15:08:31.242161 15076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:31.242283 15076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:31.242417 15076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:31.249877 15076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:31.249980 15076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:31.250082 15076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:31.250164 15076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:31.250275 15076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:31.256206 15069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } attempt: 4
I20260619 15:08:31.257436 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.256923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60120)
0619 15:08:31.257186 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.257274 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 15:08:31.258426 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.258099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60136)
0619 15:08:31.258254 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.258327 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 15:08:31.258630 15069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:39841: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
I20260619 15:08:31.261394 15076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:43431,127.24.103.125:38449,127.24.103.124:39841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.124.keytab
--principal=kudu/127.24.103.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:39841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.103.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:31.262357 14995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } attempt: 3
I20260619 15:08:31.263065 15076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:31.263453 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.262970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60146)
0619 15:08:31.263211 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.263301 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 15:08:31.264472 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.264102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60158)
0619 15:08:31.264307 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.264377 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 15:08:31.264632 15076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:31.264693 14995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:39841: Network error: Client connection negotiation failed: client connection to 127.24.103.124:39841: connect: Connection refused (error 111)
W20260619 15:08:31.276584 15084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:31.276757 15081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:31.276764 15082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:31.277730 15076 server_base.cc:1034] running on GCE node
I20260619 15:08:31.278523 15076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:31.279973 15076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:31.281195 15076 hybrid_clock.cc:648] HybridClock initialized: now 1781881711281119 us; error 48 us; skew 500 ppm
I20260619 15:08:31.281544 15076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:31 dist-test-slave-zpbw krb5kdc[14915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881711, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:31.287746 15076 init.cc:377] Logged in from keytab as kudu/127.24.103.124@KRBTEST.COM (short username kudu)
I20260619 15:08:31.289245 15076 webserver.cc:460] Webserver started at http://127.24.103.124:40137/ using document root <none> and password file <none>
I20260619 15:08:31.289956 15076 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:31.290045 15076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:31.290318 15076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:31.293448 15076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/data/instance:
uuid: "49d650aeca7c4365adaeb56a8a4e0ff2"
format_stamp: "Formatted at 2026-06-19 15:08:31 on dist-test-slave-zpbw"
server_key: "3e4c14d03c7aed6b8c412ce88333579f"
server_key_iv: "91068d519f8f9fc13724edd21805d0de"
server_key_version: "encryptionkey@0"
I20260619 15:08:31.294286 15076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/wal/instance:
uuid: "49d650aeca7c4365adaeb56a8a4e0ff2"
format_stamp: "Formatted at 2026-06-19 15:08:31 on dist-test-slave-zpbw"
server_key: "3e4c14d03c7aed6b8c412ce88333579f"
server_key_iv: "91068d519f8f9fc13724edd21805d0de"
server_key_version: "encryptionkey@0"
I20260619 15:08:31.299894 15076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 15:08:31.303541 15091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:31.305050 15076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:08:31.305243 15076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/wal
uuid: "49d650aeca7c4365adaeb56a8a4e0ff2"
format_stamp: "Formatted at 2026-06-19 15:08:31 on dist-test-slave-zpbw"
server_key: "3e4c14d03c7aed6b8c412ce88333579f"
server_key_iv: "91068d519f8f9fc13724edd21805d0de"
server_key_version: "encryptionkey@0"
I20260619 15:08:31.305483 15076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:31.327888 15076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:31.332521 15076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:31.332809 15076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:31.354048 15076 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.124:39841
I20260619 15:08:31.354070 15142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.124:39841 every 8 connection(s)
I20260619 15:08:31.355595 15076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
I20260619 15:08:31.359481 15143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:31.362005 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 15076
I20260619 15:08:31.362211 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881351219607-24989-0/minicluster-data/master-2/wal/instance
I20260619 15:08:31.362671 24989 external_mini_cluster.cc:1428] Setting key 14663efa1650c741a66b06c2a9197db5
I20260619 15:08:31.367112 15143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:08:31 dist-test-slave-zpbw krb5kdc[14915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881711, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
Jun 19 15:08:31 dist-test-slave-zpbw krb5kdc[14915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260619 15:08:31.398406 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.370304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:43431 (local address 127.0.0.1:39442)
0619 15:08:31.371279 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.371320 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:31.371486 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:31.371852 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:31.371859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:31.372666 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:31.372924 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:31.372932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.374284 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.374291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:31.375923 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:31.375940 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.376691 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.376695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:31.376813 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.383762 (+ 6949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:31.383824 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:31.387701 (+ 3877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:31.392443 (+ 4742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:31.392459 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:31.392476 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:31.393548 (+ 1072us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:31.394076 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:31.394081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:31.394084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:31.394364 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:31.395198 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:31.395210 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:31.395745 (+ 535us) client_negotiation.cc:769] Sending connection context
0619 15:08:31.396125 (+ 380us) client_negotiation.cc:240] Negotiation successful
0619 15:08:31.396720 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":453,"threads_started":1}
I20260619 15:08:31.399969 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.370270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.24.103.126:43431)
0619 15:08:31.370882 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.370890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.371428 (+ 538us) server_negotiation.cc:407] Connection header received
0619 15:08:31.371601 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.371604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.371676 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.371763 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.373078 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.374146 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.376072 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.376495 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.376595 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.388042 (+ 11447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:31.388053 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:31.388061 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:31.388120 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:31.392241 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.393733 (+ 1492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.393739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.393746 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.393917 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.394506 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.394511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.394514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.394935 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:31.395104 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:31.399042 (+ 3938us) server_negotiation.cc:299] Negotiation successful
0619 15:08:31.399330 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":249,"threads_started":1}
I20260619 15:08:31.400590 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.364751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.24.103.124:39841)
0619 15:08:31.365586 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.365598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.365640 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:08:31.367376 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.367399 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.367831 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.368185 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.370136 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.372303 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.374053 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.375120 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.375335 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.390132 (+ 14797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:31.390146 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:31.390169 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:31.390223 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:31.395044 (+ 4821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.396374 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.396385 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.396389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.396576 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.397309 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.397314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.397318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.397823 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:31.398178 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:31.399797 (+ 1619us) server_negotiation.cc:299] Negotiation successful
0619 15:08:31.400117 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":418,"threads_started":1}
I20260619 15:08:31.402678 15143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:08:31 dist-test-slave-zpbw krb5kdc[14915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881711, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
Jun 19 15:08:31 dist-test-slave-zpbw krb5kdc[14915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:08:31.420361 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.403645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:38449 (local address 127.0.0.1:50598)
0619 15:08:31.403910 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.403931 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:31.404051 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:31.404615 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:31.404618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:31.405155 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:31.405365 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:31.405377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.406903 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.406906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:31.407875 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:31.407889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.409096 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.409102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:31.409299 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.410004 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:31.410034 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:31.412923 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:31.417183 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:31.417190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:31.417194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:31.417738 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:31.418214 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:31.418221 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:31.418226 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:31.418411 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:31.419201 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:31.419209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:31.419404 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 15:08:31.419615 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 15:08:31.419944 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260619 15:08:31.420676 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.403603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.24.103.125:38449)
0619 15:08:31.403799 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.403806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.404024 (+ 218us) server_negotiation.cc:407] Connection header received
0619 15:08:31.404238 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.404244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.404335 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.404411 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.405561 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.406682 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.408042 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.408699 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.408797 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.413210 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:31.413220 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:31.413226 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:31.413279 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:31.416990 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.417861 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.417869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.417876 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.418091 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.418549 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.418557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.418562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.419056 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:31.419232 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:31.419736 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 15:08:31.420058 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 15:08:31.421896 15143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:08:31.423671 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.404030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.24.103.126:43431)
0619 15:08:31.404217 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.404225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.404315 (+ 90us) server_negotiation.cc:407] Connection header received
0619 15:08:31.404492 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.404498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.404583 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.404663 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.406334 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.407581 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.409393 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.410230 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.410364 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.415947 (+ 5583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:31.415956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:31.415962 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:31.416008 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:31.419486 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.420467 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.420475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.420478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.420651 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.421147 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.421152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.421155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.421563 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:31.421835 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:31.423055 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0619 15:08:31.423326 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 15:08:31 dist-test-slave-zpbw krb5kdc[14915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881711, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
Jun 19 15:08:31 dist-test-slave-zpbw krb5kdc[14915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260619 15:08:31.437837 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.423029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60170 (local address 127.24.103.124:39841)
0619 15:08:31.423209 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.423217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.423236 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:08:31.423385 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.423393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.423481 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.423594 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.424469 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.425741 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.427489 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.428226 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.428378 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.432419 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:31.432425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:31.432429 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:31.432467 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:31.435110 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.435878 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.435890 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.435893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.435996 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.436365 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.436370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.436373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.436654 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:31.436807 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:31.437190 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 15:08:31.437539 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":179,"server-negotiator.queue_time_us":55}
I20260619 15:08:31.437844 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.422737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60170)
0619 15:08:31.423108 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.423138 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:31.423233 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:31.423607 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:31.423612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:31.424095 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:31.424317 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:31.424330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.425859 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.425864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:31.427362 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:31.427376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.428397 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.428402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:31.428714 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.429408 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:31.429432 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:31.432195 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:31.435246 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:31.435252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:31.435257 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:31.435776 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:31.436108 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:31.436113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:31.436116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:31.436271 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:31.436768 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:31.436773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:31.436926 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 15:08:31.437095 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 15:08:31.437407 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":40}
I20260619 15:08:31.441937 14995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } attempt: 4
I20260619 15:08:31.443599 15143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2: Bootstrap starting.
I20260619 15:08:31.444628 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.425312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.24.103.125:38449)
0619 15:08:31.425486 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.425491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.425553 (+ 62us) server_negotiation.cc:407] Connection header received
0619 15:08:31.425773 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.425779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.425856 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.425944 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.427675 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.428439 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.430303 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.431054 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.431216 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.437656 (+ 6440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:31.437666 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:31.437673 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:31.437723 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:31.441223 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.441965 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.441972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.441976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.442174 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.442640 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.442645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.442648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.443070 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:31.443326 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:31.444004 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 15:08:31.444261 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 15:08:31.448165 15143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:31.449651 15143 log.cc:824] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:31.450095 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Jun 19 15:08:31 dist-test-slave-zpbw krb5kdc[14915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881710, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260619 15:08:31.454855 15143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2: No bootstrap required, opened a new log
I20260619 15:08:31.453824 14970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8e08c779082d4d4e91a5f43d12536428"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39448
I20260619 15:08:31.460896 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.442584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60174)
0619 15:08:31.442918 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.442940 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:31.443073 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:31.443421 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:31.443426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:31.443867 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:31.444116 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:31.444128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.445449 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.445454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:31.446743 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:31.446759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.447933 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.447939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:31.448191 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.449130 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:31.449147 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:31.451879 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:31.457997 (+ 6118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:31.458001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:31.458004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:31.458390 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:31.458870 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:31.458877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:31.458883 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:31.459110 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:31.459815 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:31.459820 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:31.460020 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 15:08:31.460160 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 15:08:31.460462 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260619 15:08:31.461037 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.442628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.24.103.124:39841)
0619 15:08:31.442871 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.442879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.443012 (+ 133us) server_negotiation.cc:407] Connection header received
0619 15:08:31.443191 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.443195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.443272 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.443372 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.444286 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.445300 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.447105 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.447785 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.447927 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.452109 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:31.452132 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:31.452139 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:31.452192 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:31.457835 (+ 5643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.458530 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.458536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.458540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.458715 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.459236 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.459241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.459244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.459694 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:31.459875 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:31.460316 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 15:08:31.460655 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 15:08:31.464884 15143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } }
I20260619 15:08:31.465263 15143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d650aeca7c4365adaeb56a8a4e0ff2, State: Initialized, Role: FOLLOWER
I20260619 15:08:31.465894 15143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } }
I20260619 15:08:31.466710 14995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428: Bootstrap starting.
I20260619 15:08:31.466986 15150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } } }
I20260619 15:08:31.467494 15150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:31.468222 15143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:31.470402 14995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:31.471659 14995 log.cc:824] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:31.475461 14995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428: No bootstrap required, opened a new log
I20260619 15:08:31.483404 14995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } }
I20260619 15:08:31.483825 14995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e08c779082d4d4e91a5f43d12536428, State: Initialized, Role: FOLLOWER
I20260619 15:08:31.484325 14995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } }
I20260619 15:08:31.485630 15164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } } }
I20260619 15:08:31.486258 15164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:31.486992 14995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:31.487116 14970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 15:08:31.487301 14970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [term 0 FOLLOWER]: Advancing to term 1
W20260619 15:08:31.487515 15162 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:08:31.487648 15162 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:08:31.490974 14970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } }
I20260619 15:08:31.493862 14970 leader_election.cc:288] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [CANDIDATE]: Term 1 election: Requested vote from peers 35303389fc1d4346bd10bdbe921e7e44 (127.24.103.125:38449), 49d650aeca7c4365adaeb56a8a4e0ff2 (127.24.103.124:39841)
I20260619 15:08:31.478852 15076 master_runner.cc:186] Error getting master registration for 127.24.103.126:43431: OK, instance_id { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" instance_seqno: 1781881710298045 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 15:08:31.494714 15118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e08c779082d4d4e91a5f43d12536428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2"
I20260619 15:08:31.495337 15118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:31.500149 15118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e08c779082d4d4e91a5f43d12536428 in term 1.
I20260619 15:08:31.501168 14946 leader_election.cc:302] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d650aeca7c4365adaeb56a8a4e0ff2, 8e08c779082d4d4e91a5f43d12536428; no voters:
I20260619 15:08:31.502056 15164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:31.494671 15076 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 15:08:31.509186 15164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [term 1 LEADER]: Becoming Leader. State: Replica: 8e08c779082d4d4e91a5f43d12536428, State: Running, Role: LEADER
I20260619 15:08:31.510260 15164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } }
I20260619 15:08:31.526113 15168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e08c779082d4d4e91a5f43d12536428. Latest consensus state: current_term: 1 leader_uuid: "8e08c779082d4d4e91a5f43d12536428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } } }
I20260619 15:08:31.526738 15168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:31.528551 15174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260619 15:08:31.537441 14949 tablet.cc:2369] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:08:31.548086 15174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:31.558477 15069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } attempt: 5
W20260619 15:08:31.585280 15097 tablet.cc:2369] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:08:31.586278 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.532311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.24.103.124:39841)
0619 15:08:31.532519 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.532525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.539964 (+ 7439us) server_negotiation.cc:407] Connection header received
0619 15:08:31.541054 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.541061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.541144 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.541222 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.549901 (+ 8679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.551096 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.558798 (+ 7702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.559547 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.559671 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.574505 (+ 14834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:31.574514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:31.574519 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:31.574568 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:31.578510 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.580541 (+ 2031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.580548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.580555 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.580750 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.584180 (+ 3430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.584187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.584191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.584591 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:31.584869 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:31.585561 (+ 692us) server_negotiation.cc:299] Negotiation successful
0619 15:08:31.585863 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:08:31.587976 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.533074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.24.103.125:38449)
0619 15:08:31.533264 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.533273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.539379 (+ 6106us) server_negotiation.cc:407] Connection header received
0619 15:08:31.539747 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.539757 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.539877 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.540030 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.544715 (+ 4685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.545828 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.549086 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.549738 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.549856 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.557241 (+ 7385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:31.557252 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:31.557260 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:31.557322 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:31.564215 (+ 6893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.575919 (+ 11704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.575929 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.575935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.576174 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.579433 (+ 3259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.579443 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.579449 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.580041 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:31.580379 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:31.587343 (+ 6964us) server_negotiation.cc:299] Negotiation successful
0619 15:08:31.587659 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 15:08:31.595568 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.536030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.24.103.126:43431)
0619 15:08:31.536239 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.536245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.544156 (+ 7911us) server_negotiation.cc:407] Connection header received
0619 15:08:31.547306 (+ 3150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.547313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.547401 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.547475 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.555381 (+ 7906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.556591 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.569589 (+ 12998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.570338 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.570457 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.581346 (+ 10889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:31.581354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:31.581361 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:31.581410 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:31.585889 (+ 4479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.587494 (+ 1605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.587500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.587505 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.591867 (+ 4362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.592371 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.592376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.592379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.592858 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:31.593178 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:31.594857 (+ 1679us) server_negotiation.cc:299] Negotiation successful
0619 15:08:31.595122 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 15:08:31.601460 15118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [term 1 FOLLOWER]: Refusing update from remote peer 8e08c779082d4d4e91a5f43d12536428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Jun 19 15:08:31 dist-test-slave-zpbw krb5kdc[14915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881710, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260619 15:08:31.603440 15168 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:08:31.614675 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.559168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:39841 (local address 127.0.0.1:60184)
0619 15:08:31.559609 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:31.559632 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:31.559750 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:31.590700 (+ 30950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:31.590707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:31.591275 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:31.591499 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:31.591512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.595863 (+ 4351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.595873 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:31.597287 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:31.597309 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.598357 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.598364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:31.598653 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.599455 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:31.599479 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:31.606288 (+ 6809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:31.611395 (+ 5107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:31.611404 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:31.611410 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:31.612028 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:31.612614 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:31.612623 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:31.612628 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:31.612878 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:31.613634 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:31.613639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:31.613806 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 15:08:31.613955 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 15:08:31.614290 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260619 15:08:31.616199 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:31.559989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.24.103.124:39841)
0619 15:08:31.589325 (+ 29336us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.589333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.589363 (+ 30us) server_negotiation.cc:407] Connection header received
0619 15:08:31.589413 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.589417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.589503 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.594071 (+ 4568us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.594403 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.595678 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.597516 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:31.598214 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:31.598332 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:31.606657 (+ 8325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:31.606664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:31.606668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:31.606707 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:31.610876 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.612234 (+ 1358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.612246 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.612252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.612465 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:31.613053 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:31.613060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:31.613066 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:31.613489 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:31.615402 (+ 1913us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:31.615533 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 15:08:31.615776 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26446,"thread_start_us":22763,"threads_started":1}
I20260619 15:08:31.623510 15069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44: Bootstrap starting.
I20260619 15:08:31.629120 15069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:31.630659 15069 log.cc:824] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:31.634471 15150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e08c779082d4d4e91a5f43d12536428. Latest consensus state: current_term: 1 leader_uuid: "8e08c779082d4d4e91a5f43d12536428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } } }
I20260619 15:08:31.635811 15150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:31.638199 15180 mvcc.cc:205] Tried to move back new op lower bound from 7298587490685603840 to 7298587490354167808. Current Snapshot: MvccSnapshot[applied={T|T < 7298587490685603840}]
I20260619 15:08:31.639150 15168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e08c779082d4d4e91a5f43d12536428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } } }
I20260619 15:08:31.639427 15168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:31.640053 15184 mvcc.cc:205] Tried to move back new op lower bound from 7298587490685603840 to 7298587490354167808. Current Snapshot: MvccSnapshot[applied={T|T < 7298587490685603840}]
I20260619 15:08:31.643467 15069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44: No bootstrap required, opened a new log
I20260619 15:08:31.657053 15069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } }
I20260619 15:08:31.657575 15069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35303389fc1d4346bd10bdbe921e7e44, State: Initialized, Role: FOLLOWER
I20260619 15:08:31.658243 15069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } }
I20260619 15:08:31.658273 15150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e08c779082d4d4e91a5f43d12536428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } } }
I20260619 15:08:31.658535 15150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:31.660388 15174 catalog_manager.cc:1347] Generated new cluster ID: 3b80776afe584386a5a3d61aa61e3833
I20260619 15:08:31.660526 15174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:31.665970 15193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } } }
I20260619 15:08:31.668493 15193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:31.668357 15069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:31.668919 15043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:31.669160 15044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e08c779082d4d4e91a5f43d12536428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "35303389fc1d4346bd10bdbe921e7e44"
I20260619 15:08:31.673712 15043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [term 1 FOLLOWER]: Refusing update from remote peer 8e08c779082d4d4e91a5f43d12536428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:08:31.677400 15044 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e08c779082d4d4e91a5f43d12536428 in term 1.
I20260619 15:08:31.679630 15164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:08:31.695674 14928 master_runner.cc:186] Error getting master registration for 127.24.103.125:38449: OK, instance_id { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" instance_seqno: 1781881710817167 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 15:08:31.736917 14928 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 15:08:31.760906 15174 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:31.764076 15174 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:31.778573 15193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e08c779082d4d4e91a5f43d12536428. Latest consensus state: current_term: 1 leader_uuid: "8e08c779082d4d4e91a5f43d12536428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } } }
I20260619 15:08:31.778927 15193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:31.780133 15168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e08c779082d4d4e91a5f43d12536428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } } }
I20260619 15:08:31.780416 15168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:31.788936 15201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e08c779082d4d4e91a5f43d12536428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e08c779082d4d4e91a5f43d12536428" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43431 } } peers { permanent_uuid: "35303389fc1d4346bd10bdbe921e7e44" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 38449 } } peers { permanent_uuid: "49d650aeca7c4365adaeb56a8a4e0ff2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 39841 } } }
I20260619 15:08:31.789870 15201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:31.790380 15174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e08c779082d4d4e91a5f43d12536428: Generated new TSK 0
I20260619 15:08:31.792058 15174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:31.835393 15209 catalog_manager.cc:1259] Loaded cluster ID: 3b80776afe584386a5a3d61aa61e3833
I20260619 15:08:31.835544 15209 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44: loading cluster ID for follower catalog manager: success
I20260619 15:08:31.839141 15209 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44: acquiring CA information for follower catalog manager: success
I20260619 15:08:31.841866 15209 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 35303389fc1d4346bd10bdbe921e7e44: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 15:08:31.972565 15073 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:31.953205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.24.103.125:38449)
0619 15:08:31.953421 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.953430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.969253 (+ 15823us) server_negotiation.cc:407] Connection header received
0619 15:08:31.970101 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.970111 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.970264 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.970433 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.972264 (+ 1831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50612: BlockingRecv error: recv got EOF from 127.0.0.1:50612 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 15:08:31.975279 15147 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:31.953854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.24.103.126:43431)
0619 15:08:31.954145 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.954154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.973298 (+ 19144us) server_negotiation.cc:407] Connection header received
0619 15:08:31.973611 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.973619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.973706 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.973791 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.974940 (+ 1149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39472: BlockingRecv error: recv got EOF from 127.0.0.1:39472 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 15:08:31.980392 15182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:31.960182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.24.103.124:39841)
0619 15:08:31.960394 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:31.960403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:31.978236 (+ 17833us) server_negotiation.cc:407] Connection header received
0619 15:08:31.978740 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:31.978748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:31.978887 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:31.978963 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:31.980135 (+ 1172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60200: BlockingRecv error: recv got EOF from 127.0.0.1:60200 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 15:08:32.051501 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.027718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.24.103.125:38449)
0619 15:08:32.027915 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.027921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.043506 (+ 15585us) server_negotiation.cc:407] Connection header received
0619 15:08:32.043797 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.043804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.044045 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.044223 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.044942 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.046179 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.048265 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.048969 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.049087 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.049849 (+ 762us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.050626 (+ 777us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.050748 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.051028 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.051143 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 15:08:32.097564 15182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:32.088019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.24.103.124:39841)
0619 15:08:32.088214 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.088221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.095514 (+ 7293us) server_negotiation.cc:407] Connection header received
0619 15:08:32.095789 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.095797 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.095954 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.096095 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:32.097284 (+ 1189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60208: BlockingRecv error: recv got EOF from 127.0.0.1:60208 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 15:08:32.146286 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.135576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.24.103.125:38449)
0619 15:08:32.135795 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.135802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.139544 (+ 3742us) server_negotiation.cc:407] Connection header received
0619 15:08:32.139757 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.139763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.139860 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.139973 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.140513 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.141657 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.143963 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.144695 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.144827 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.145380 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.145536 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.145632 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.145826 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.145916 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 15:08:32.164995 15182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:32.161685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.24.103.124:39841)
0619 15:08:32.161862 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.161868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.163408 (+ 1540us) server_negotiation.cc:407] Connection header received
0619 15:08:32.163607 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.163614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.163717 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.163828 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:32.164759 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60218: BlockingRecv error: recv got EOF from 127.0.0.1:60218 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 15:08:32.198331 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.182812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.24.103.125:38449)
0619 15:08:32.183009 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.183015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.191543 (+ 8528us) server_negotiation.cc:407] Connection header received
0619 15:08:32.191768 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.191774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.191879 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.191991 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.192503 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.193700 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.195758 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.196450 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.196583 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.197068 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.197222 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.197316 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.197903 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.197992 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 15:08:32.213788 15182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:32.211000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.24.103.124:39841)
0619 15:08:32.211176 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.211182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.212284 (+ 1102us) server_negotiation.cc:407] Connection header received
0619 15:08:32.212459 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.212466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.212569 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.212674 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:32.213548 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60232: BlockingRecv error: recv got EOF from 127.0.0.1:60232 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 15:08:32.228245 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.222319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.24.103.125:38449)
0619 15:08:32.222492 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.222497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.222871 (+ 374us) server_negotiation.cc:407] Connection header received
0619 15:08:32.223090 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.223095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.223188 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.223266 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.223715 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.224445 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.226398 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.226969 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.227069 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.227548 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.227656 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.227731 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.227891 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.227958 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 15:08:32.236857 15182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:32.234916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.24.103.124:39841)
0619 15:08:32.235177 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.235183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.235432 (+ 249us) server_negotiation.cc:407] Connection header received
0619 15:08:32.235606 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.235610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.235687 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.235772 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:32.236626 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60234: BlockingRecv error: recv got EOF from 127.0.0.1:60234 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
I20260619 15:08:32.255481 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.248851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.24.103.125:38449)
0619 15:08:32.249095 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.249101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.249462 (+ 361us) server_negotiation.cc:407] Connection header received
0619 15:08:32.249683 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.249687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.249765 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.249848 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.250411 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.251321 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.253415 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.253915 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.254019 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.254614 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.254757 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.254907 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.255094 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.255168 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260619 15:08:32.264381 15182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:32.262102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.24.103.124:39841)
0619 15:08:32.262344 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.262350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.262807 (+ 457us) server_negotiation.cc:407] Connection header received
0619 15:08:32.263023 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.263027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.263104 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.263202 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:32.264145 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60238: BlockingRecv error: recv got EOF from 127.0.0.1:60238 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 15:08:32.296757 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.290088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.24.103.125:38449)
0619 15:08:32.290325 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.290332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.290752 (+ 420us) server_negotiation.cc:407] Connection header received
0619 15:08:32.291009 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.291016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.291121 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.291250 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.291646 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.292849 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.294638 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.295388 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.295546 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.295962 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.296123 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.296224 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.296332 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.296429 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260619 15:08:32.309078 15182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:32.303817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.24.103.124:39841)
0619 15:08:32.304018 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.304026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.307416 (+ 3390us) server_negotiation.cc:407] Connection header received
0619 15:08:32.307641 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.307647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.307758 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.307856 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:32.308845 (+ 989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60254: BlockingRecv error: recv got EOF from 127.0.0.1:60254 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:08:32.351673 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.345355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50694 (local address 127.24.103.125:38449)
0619 15:08:32.345597 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.345604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.346070 (+ 466us) server_negotiation.cc:407] Connection header received
0619 15:08:32.346292 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.346296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.346365 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.346444 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.346936 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.347700 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.349706 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.350170 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.350264 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.350874 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.350983 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.351059 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.351246 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.351315 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 15:08:32.370519 15182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:32.362817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.24.103.124:39841)
0619 15:08:32.363006 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.363012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.368619 (+ 5607us) server_negotiation.cc:407] Connection header received
0619 15:08:32.368954 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.368963 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.369104 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.369236 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:32.370255 (+ 1019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60270: BlockingRecv error: recv got EOF from 127.0.0.1:60270 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 15:08:32.444417 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.438930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.24.103.125:38449)
0619 15:08:32.439151 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.439158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.439544 (+ 386us) server_negotiation.cc:407] Connection header received
0619 15:08:32.439808 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.439813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.439903 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.440005 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.440457 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.441222 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.442604 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.443091 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.443259 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.443680 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.443785 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.443887 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.444045 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.444130 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 15:08:32.452587 15182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:32.450492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.24.103.124:39841)
0619 15:08:32.450732 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.450740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.451068 (+ 328us) server_negotiation.cc:407] Connection header received
0619 15:08:32.451309 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.451315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.451400 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.451550 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:32.452363 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60284: BlockingRecv error: recv got EOF from 127.0.0.1:60284 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 15:08:32.489641 15162 catalog_manager.cc:1259] Loaded cluster ID: 3b80776afe584386a5a3d61aa61e3833
I20260619 15:08:32.489742 15162 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2: loading cluster ID for follower catalog manager: success
I20260619 15:08:32.492272 15162 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2: acquiring CA information for follower catalog manager: success
I20260619 15:08:32.494059 15162 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 49d650aeca7c4365adaeb56a8a4e0ff2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 15:08:32.592242 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.586632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.24.103.125:38449)
0619 15:08:32.586938 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.586944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.587214 (+ 270us) server_negotiation.cc:407] Connection header received
0619 15:08:32.587474 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.587482 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.587597 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.587708 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.588180 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.588966 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.590349 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.590835 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.590979 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.591436 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.591559 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.591662 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.591821 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.591906 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 15:08:32.603860 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.598156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.24.103.124:39841)
0619 15:08:32.598367 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.598375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.598706 (+ 331us) server_negotiation.cc:407] Connection header received
0619 15:08:32.598924 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.598928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.599101 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.599208 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.599607 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.600396 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.601657 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.602063 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.602153 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.602648 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.603303 (+ 655us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.603412 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.603539 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.603623 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 15:08:32.620093 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.610854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.24.103.125:38449)
0619 15:08:32.611124 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.611134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.612996 (+ 1862us) server_negotiation.cc:407] Connection header received
0619 15:08:32.613307 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.613316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.613443 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.613565 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.614397 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.615477 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.617160 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.617845 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.617968 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.618613 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.618726 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.618878 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.619683 (+ 805us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.619781 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 15:08:32.621048 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.610999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39478 (local address 127.24.103.126:43431)
0619 15:08:32.613096 (+ 2097us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.613102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.613121 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:08:32.613244 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.613249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.613421 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.613603 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.614246 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.615457 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.617569 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.617988 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.618134 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.619038 (+ 904us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.619500 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.619586 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.620713 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.620802 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1921,"thread_start_us":1778,"threads_started":1}
I20260619 15:08:32.621596 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:32.611281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.24.103.124:39841)
0619 15:08:32.611490 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:32.611498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:32.614996 (+ 3498us) server_negotiation.cc:407] Connection header received
0619 15:08:32.615161 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:32.615166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:32.615252 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:32.615342 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:32.616092 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.617159 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.619035 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:32.619583 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:32.619722 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:32.620173 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:32.620358 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:32.620461 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:32.621259 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 15:08:32.621337 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:08:32.626282 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 14928
I20260619 15:08:32.627456 14990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:08:32.738945 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 14928
W20260619 15:08:32.759442 15093 connection.cc:535] server connection from 127.0.0.1:60174 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 15:08:32.759881 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 15002
I20260619 15:08:32.760982 15064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:08:32.869138 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 15002
I20260619 15:08:32.892303 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 15076
I20260619 15:08:32.893436 15138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:08:33.001686 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 15076
2026-06-19T15:08:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4533 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:08:33 dist-test-slave-zpbw krb5kdc[15323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:08:33 dist-test-slave-zpbw krb5kdc[15323](info): set up 2 sockets
Jun 19 15:08:33 dist-test-slave-zpbw krb5kdc[15323](info): commencing operation
krb5kdc: starting...
W20260619 15:08:34.115823 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/test-user.keytab.
Jun 19 15:08:34 dist-test-slave-zpbw krb5kdc[15323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:08:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:08:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260619 15:08:34.387902 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35851
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:35851,127.24.103.125:34055,127.24.103.124:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:08:34.763648 15336 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 15:08:34.764052 15336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:34.764201 15336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:34.764322 15336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:34.771872 15336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:34.771989 15336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:34.772102 15336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:34.772209 15336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:34.772321 15336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:34.782851 15336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:35851,127.24.103.125:34055,127.24.103.124:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:34.784271 15336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:34.785632 15336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:34.798151 15341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:34.798132 15344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:34.798084 15342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:34.799217 15336 server_base.cc:1034] running on GCE node
I20260619 15:08:34.799984 15336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:34.801458 15336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:34.802809 15336 hybrid_clock.cc:648] HybridClock initialized: now 1781881714802696 us; error 49 us; skew 500 ppm
I20260619 15:08:34.803174 15336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:34 dist-test-slave-zpbw krb5kdc[15323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881714, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:34.806473 15336 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260619 15:08:34.808028 15336 webserver.cc:460] Webserver started at http://127.24.103.126:39209/ using document root <none> and password file <none>
I20260619 15:08:34.808799 15336 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:34.808952 15336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:34.809289 15336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:34.812363 15336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/data/instance:
uuid: "33e6636beff74721aeaec12eca25d3f0"
format_stamp: "Formatted at 2026-06-19 15:08:34 on dist-test-slave-zpbw"
server_key: "2e8079d0c2c7977171dcca37bc0648fa"
server_key_iv: "dc83e947ed9b77fa12d5f1a558ccae1f"
server_key_version: "encryptionkey@0"
I20260619 15:08:34.813140 15336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/wal/instance:
uuid: "33e6636beff74721aeaec12eca25d3f0"
format_stamp: "Formatted at 2026-06-19 15:08:34 on dist-test-slave-zpbw"
server_key: "2e8079d0c2c7977171dcca37bc0648fa"
server_key_iv: "dc83e947ed9b77fa12d5f1a558ccae1f"
server_key_version: "encryptionkey@0"
I20260619 15:08:34.818480 15336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 15:08:34.822181 15351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:34.823637 15336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:08:34.823835 15336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/wal
uuid: "33e6636beff74721aeaec12eca25d3f0"
format_stamp: "Formatted at 2026-06-19 15:08:34 on dist-test-slave-zpbw"
server_key: "2e8079d0c2c7977171dcca37bc0648fa"
server_key_iv: "dc83e947ed9b77fa12d5f1a558ccae1f"
server_key_version: "encryptionkey@0"
I20260619 15:08:34.824076 15336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:34.836230 15336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:34.840479 15336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:34.840790 15336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:34.860144 15336 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:35851
I20260619 15:08:34.860183 15402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:35851 every 8 connection(s)
I20260619 15:08:34.861641 15336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/data/info.pb
I20260619 15:08:34.865918 15403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:34.870560 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 15336
I20260619 15:08:34.870914 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-0/wal/instance
I20260619 15:08:34.871397 24989 external_mini_cluster.cc:1428] Setting key 04aa53fae8edbd5b5bf6e01d962c62d0
I20260619 15:08:34.873023 15403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:08:34 dist-test-slave-zpbw krb5kdc[15323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881714, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.125@KRBTEST.COM; defaulting to no policy
I20260619 15:08:34.904392 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:34.876573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35851 (local address 127.0.0.1:50140)
0619 15:08:34.877602 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:34.877662 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:34.880765 (+ 3103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:34.882192 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:34.882239 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:34.883404 (+ 1165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:34.884120 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:34.884151 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:34.886708 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:34.886714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:34.888555 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:34.888577 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:34.890444 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:34.890454 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:34.890658 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:34.892052 (+ 1394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:34.892097 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:34.895260 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:34.899393 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:34.899407 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:34.899434 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:34.899876 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:34.900390 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:34.900398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:34.900402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:34.900589 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:34.901569 (+ 980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:34.901578 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:34.901834 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 15:08:34.902112 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 15:08:34.902775 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":149,"thread_start_us":407,"threads_started":1}
I20260619 15:08:34.904412 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:34.876573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.24.103.126:35851)
0619 15:08:34.878749 (+ 2176us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:34.878756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:34.878861 (+ 105us) server_negotiation.cc:407] Connection header received
0619 15:08:34.881392 (+ 2531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:34.881404 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:34.881816 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:34.882082 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:34.884438 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:34.886491 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:34.888777 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:34.889764 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:34.890005 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:34.895580 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:34.895590 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:34.895606 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:34.895648 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:34.899186 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:34.900090 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:34.900097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:34.900100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:34.900265 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:34.900715 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:34.900727 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:34.900730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:34.901401 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:34.901663 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:34.902251 (+ 588us) server_negotiation.cc:299] Negotiation successful
0619 15:08:34.902612 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1966,"spinlock_wait_cycles":17280,"thread_start_us":1707,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.125@KRBTEST.COM" created.
I20260619 15:08:34.908181 15403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:08:34.909394 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:34.908814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60072)
0619 15:08:34.909050 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:34.909209 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 15:08:34.909731 15354 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
I20260619 15:08:34.911518 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:34.911015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60088)
0619 15:08:34.911214 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:34.911318 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 15:08:34.911757 15403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:34055: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.125.keytab.
Entry for principal kudu/127.24.103.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.125.keytab.
I20260619 15:08:34.935886 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:34055
--webserver_interface=127.24.103.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.125.keytab
--principal=kudu/127.24.103.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:35851,127.24.103.125:34055,127.24.103.124:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:08:34.961156 15403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } attempt: 1
I20260619 15:08:34.962361 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:34.961879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60100)
0619 15:08:34.962106 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:34.962228 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 15:08:34.963444 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:34.963092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60112)
0619 15:08:34.963240 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:34.963317 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 15:08:34.963634 15403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:34055: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
I20260619 15:08:35.031911 15403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } attempt: 2
I20260619 15:08:35.033249 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.032603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60116)
0619 15:08:35.032897 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.033038 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 15:08:35.034505 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.034118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60126)
0619 15:08:35.034288 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.034368 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 15:08:35.034734 15403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:34055: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
I20260619 15:08:35.126044 15403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } attempt: 3
I20260619 15:08:35.127400 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.126748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60130)
0619 15:08:35.127044 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.127196 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 15:08:35.128636 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.128115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60134)
0619 15:08:35.128338 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.128429 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 15:08:35.128861 15403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:34055: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
I20260619 15:08:35.272153 15403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } attempt: 4
I20260619 15:08:35.273536 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.272883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60136)
0619 15:08:35.273238 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.273366 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260619 15:08:35.274636 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.274206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60138)
0619 15:08:35.274365 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.274467 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 15:08:35.274909 15403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:34055: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34055: connect: Connection refused (error 111)
W20260619 15:08:35.287179 15410 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 15:08:35.287586 15410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:35.287709 15410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:35.287844 15410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:08:35.295454 15410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:35.295608 15410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:35.295724 15410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:35.295818 15410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:35.295933 15410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:35.306604 15410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:35851,127.24.103.125:34055,127.24.103.124:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.125.keytab
--principal=kudu/127.24.103.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:34055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.103.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:35.308019 15410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:35.309401 15410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:35.321542 15415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:35.321645 15416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:35.321645 15418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:35.322919 15410 server_base.cc:1034] running on GCE node
I20260619 15:08:35.323764 15410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:35.325366 15410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:35.326634 15410 hybrid_clock.cc:648] HybridClock initialized: now 1781881715326569 us; error 46 us; skew 500 ppm
I20260619 15:08:35.327039 15410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:35 dist-test-slave-zpbw krb5kdc[15323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881715, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:35.330421 15410 init.cc:377] Logged in from keytab as kudu/127.24.103.125@KRBTEST.COM (short username kudu)
I20260619 15:08:35.332185 15410 webserver.cc:460] Webserver started at http://127.24.103.125:33059/ using document root <none> and password file <none>
I20260619 15:08:35.332949 15410 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:35.333096 15410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:35.333442 15410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:35.336689 15410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/data/instance:
uuid: "25fcf70d700f4a13865db2376045beb0"
format_stamp: "Formatted at 2026-06-19 15:08:35 on dist-test-slave-zpbw"
server_key: "2dc5d663ac18a0a6b03edf6a38ca4d19"
server_key_iv: "408520059fad433491bf2c167b7b342f"
server_key_version: "encryptionkey@0"
I20260619 15:08:35.337548 15410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/wal/instance:
uuid: "25fcf70d700f4a13865db2376045beb0"
format_stamp: "Formatted at 2026-06-19 15:08:35 on dist-test-slave-zpbw"
server_key: "2dc5d663ac18a0a6b03edf6a38ca4d19"
server_key_iv: "408520059fad433491bf2c167b7b342f"
server_key_version: "encryptionkey@0"
I20260619 15:08:35.343192 15410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 15:08:35.346992 15425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:35.348511 15410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:08:35.348718 15410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/wal
uuid: "25fcf70d700f4a13865db2376045beb0"
format_stamp: "Formatted at 2026-06-19 15:08:35 on dist-test-slave-zpbw"
server_key: "2dc5d663ac18a0a6b03edf6a38ca4d19"
server_key_iv: "408520059fad433491bf2c167b7b342f"
server_key_version: "encryptionkey@0"
I20260619 15:08:35.348974 15410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:35.368243 15410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:35.372584 15410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:35.372898 15410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:35.393755 15410 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.125:34055
I20260619 15:08:35.393776 15476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.125:34055 every 8 connection(s)
I20260619 15:08:35.395321 15410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/data/info.pb
I20260619 15:08:35.399220 15477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:35.403924 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 15410
I20260619 15:08:35.404150 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-1/wal/instance
I20260619 15:08:35.404579 24989 external_mini_cluster.cc:1428] Setting key 07effc4986328a8c9a14f54012e06733
I20260619 15:08:35.406509 15477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:08:35 dist-test-slave-zpbw krb5kdc[15323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881715, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.124@KRBTEST.COM" created.
I20260619 15:08:35.434739 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.409566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.24.103.126:35851)
0619 15:08:35.418515 (+ 8949us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:35.418520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:35.418537 (+ 17us) server_negotiation.cc:407] Connection header received
0619 15:08:35.418591 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:35.418595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:35.418662 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:35.418836 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:35.420764 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.421587 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.423327 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.423758 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.423903 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.428136 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:35.428142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:35.428145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:35.428188 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:35.431145 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.432072 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.432079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.432083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.432257 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.432684 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.432751 (+ 67us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.432757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.433058 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:35.433220 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:35.434053 (+ 833us) server_negotiation.cc:299] Negotiation successful
0619 15:08:35.434352 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8831,"thread_start_us":8705,"threads_started":1}
I20260619 15:08:35.436092 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.410341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35851 (local address 127.0.0.1:50150)
0619 15:08:35.415755 (+ 5414us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.415825 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:35.417785 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:35.419211 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:35.419222 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:35.419859 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:35.420490 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:35.420512 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.421782 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.421787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:35.423150 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:35.423165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.424017 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.424024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:35.424279 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.425054 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:35.425079 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:35.427805 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:35.431328 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:35.431338 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:35.431361 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:35.431855 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:35.432381 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:35.432385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:35.432387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:35.432571 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:35.433190 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:35.433196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:35.433666 (+ 470us) client_negotiation.cc:769] Sending connection context
0619 15:08:35.433918 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 15:08:35.434460 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5021,"thread_start_us":4725,"threads_started":1}
I20260619 15:08:35.438201 15477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:08:35 dist-test-slave-zpbw krb5kdc[15323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881715, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260619 15:08:35.453685 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.439053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60148)
0619 15:08:35.439222 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.439247 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:35.439357 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:35.440438 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:35.440442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:35.440906 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:35.441043 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:35.441052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.442590 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.442599 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:35.443563 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:35.443576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.444437 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.444442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:35.444681 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.445317 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:35.445334 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:35.447643 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:35.451040 (+ 3397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:35.451046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:35.451050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:35.451427 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:35.451841 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:35.451845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:35.451849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:35.452006 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:35.452631 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:35.452638 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:35.452800 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 15:08:35.452952 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 15:08:35.453267 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":49}
I20260619 15:08:35.453776 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.439315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.24.103.125:34055)
0619 15:08:35.440000 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:35.440006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:35.440033 (+ 27us) server_negotiation.cc:407] Connection header received
0619 15:08:35.440082 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:35.440088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:35.440271 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:35.440456 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:35.441218 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.442425 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.443717 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.444275 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.444426 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.447869 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:35.447882 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:35.447897 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:35.447924 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:35.450898 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.451564 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.451576 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.451584 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.451707 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.452107 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.452113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.452119 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.452502 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:35.452731 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:35.453101 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 15:08:35.453396 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":531,"thread_start_us":392,"threads_started":1}
I20260619 15:08:35.456321 15477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:08:35.457643 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.457149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54846)
0619 15:08:35.457362 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.457470 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 15:08:35.457926 15427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
I20260619 15:08:35.459704 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.459294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54852)
0619 15:08:35.459485 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.459580 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 15:08:35.459957 15477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:36067: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.124.keytab.
Entry for principal kudu/127.24.103.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.124.keytab.
I20260619 15:08:35.466722 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn730WE/build/asan/bin/kudu
/tmp/dist-test-taskn730WE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:36067
--webserver_interface=127.24.103.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.124.keytab
--principal=kudu/127.24.103.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:35851,127.24.103.125:34055,127.24.103.124:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:08:35.509354 15477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } attempt: 1
I20260619 15:08:35.510599 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.510103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54864)
0619 15:08:35.510328 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.510443 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 15:08:35.511658 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.511287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54870)
0619 15:08:35.511464 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.511542 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 15:08:35.511869 15477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:36067: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
I20260619 15:08:35.574163 15403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } attempt: 5
I20260619 15:08:35.580084 15477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } attempt: 2
I20260619 15:08:35.581168 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.580679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54880)
0619 15:08:35.580922 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.581008 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 15:08:35.582252 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.581848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54884)
0619 15:08:35.582006 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.582095 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 15:08:35.582502 15477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:36067: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Jun 19 15:08:35 dist-test-slave-zpbw krb5kdc[15323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881714, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260619 15:08:35.589991 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.574996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60156)
0619 15:08:35.575229 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.575251 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:35.575405 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:35.575800 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:35.575805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:35.576231 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:35.576407 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:35.576443 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.577930 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.577933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:35.578839 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:35.578859 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.580057 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.580063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:35.580305 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.581462 (+ 1157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:35.581496 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:35.583822 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:35.587688 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:35.587693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:35.587696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:35.588072 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:35.588483 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:35.588486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:35.588488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:35.588617 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:35.589247 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:35.589250 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:35.589386 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 15:08:35.589508 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 15:08:35.589744 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 15:08:35.590159 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.575077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60156 (local address 127.24.103.125:34055)
0619 15:08:35.575250 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:35.575257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:35.575333 (+ 76us) server_negotiation.cc:407] Connection header received
0619 15:08:35.575559 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:35.575565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:35.575648 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:35.575762 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:35.576625 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.577780 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.579011 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.579872 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.580036 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.584017 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:35.584023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:35.584028 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:35.584066 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:35.587522 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.588214 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.588220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.588223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.588371 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.588723 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.588727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.588730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.589129 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:35.589280 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:35.589646 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 15:08:35.589872 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:08:35.591238 15403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:08:35.592290 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.591924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54892)
0619 15:08:35.592088 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.592170 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 15:08:35.593240 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.592906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54900)
0619 15:08:35.593050 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.593144 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 15:08:35.593436 15403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:36067: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
I20260619 15:08:35.644758 15403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } attempt: 1
I20260619 15:08:35.645967 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.645492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54904)
0619 15:08:35.645698 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.645799 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 15:08:35.647042 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.646684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54908)
0619 15:08:35.646871 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.646937 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 15:08:35.647259 15403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:36067: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
I20260619 15:08:35.673782 15477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } attempt: 3
I20260619 15:08:35.675093 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.674564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54916)
0619 15:08:35.674772 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.674934 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 15:08:35.676208 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.675773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54928)
0619 15:08:35.676005 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.676085 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260619 15:08:35.676401 15477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:36067: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
I20260619 15:08:35.715498 15403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } attempt: 2
I20260619 15:08:35.716684 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.716188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54938)
0619 15:08:35.716438 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.716523 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 15:08:35.717794 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.717461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54940)
0619 15:08:35.717622 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.717688 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 15:08:35.718003 15403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:36067: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
W20260619 15:08:35.811611 15485 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 15:08:35.812012 15485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:08:35.812134 15485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:08:35.812266 15485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 15:08:35.819638 15477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } attempt: 4
W20260619 15:08:35.820266 15485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:08:35.820387 15485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:08:35.820508 15485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:08:35.820588 15485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:08:35.820708 15485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:08:35.820982 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.820360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54956)
0619 15:08:35.820673 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.820810 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 15:08:35.822064 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.821720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54970)
0619 15:08:35.821876 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.821957 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 15:08:35.822328 15477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:36067: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
I20260619 15:08:35.824190 15403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } attempt: 3
I20260619 15:08:35.825169 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.824740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54984)
0619 15:08:35.824914 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.825005 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 15:08:35.826192 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.825821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:54992)
0619 15:08:35.826007 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.826088 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260619 15:08:35.826385 15403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:36067: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36067: connect: Connection refused (error 111)
I20260619 15:08:35.831961 15485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:35851,127.24.103.125:34055,127.24.103.124:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/krb5kdc/kudu_127.24.103.124.keytab
--principal=kudu/127.24.103.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:36067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.103.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 15:08:35.833356 15485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:08:35.834728 15485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:08:35.846369 15490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:35.846520 15493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:08:35.846369 15491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:08:35.847402 15485 server_base.cc:1034] running on GCE node
I20260619 15:08:35.848174 15485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:08:35.849570 15485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:08:35.850850 15485 hybrid_clock.cc:648] HybridClock initialized: now 1781881715850713 us; error 53 us; skew 500 ppm
I20260619 15:08:35.851200 15485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 15:08:35 dist-test-slave-zpbw krb5kdc[15323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881715, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:08:35.855114 15485 init.cc:377] Logged in from keytab as kudu/127.24.103.124@KRBTEST.COM (short username kudu)
I20260619 15:08:35.856768 15485 webserver.cc:460] Webserver started at http://127.24.103.124:46697/ using document root <none> and password file <none>
I20260619 15:08:35.857553 15485 fs_manager.cc:359] Metadata directory not provided
I20260619 15:08:35.857697 15485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:08:35.858057 15485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:08:35.861647 15485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/data/instance:
uuid: "7484f8dfdf4f4185ace4afd7fab343f2"
format_stamp: "Formatted at 2026-06-19 15:08:35 on dist-test-slave-zpbw"
server_key: "42a85ca2d9f730d462ff762db349ffba"
server_key_iv: "a925c02f2440ec8c2b69436897f8298d"
server_key_version: "encryptionkey@0"
I20260619 15:08:35.862622 15485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/wal/instance:
uuid: "7484f8dfdf4f4185ace4afd7fab343f2"
format_stamp: "Formatted at 2026-06-19 15:08:35 on dist-test-slave-zpbw"
server_key: "42a85ca2d9f730d462ff762db349ffba"
server_key_iv: "a925c02f2440ec8c2b69436897f8298d"
server_key_version: "encryptionkey@0"
I20260619 15:08:35.869056 15485 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 15:08:35.873103 15500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:08:35.874516 15485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:08:35.874732 15485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/data,/tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/wal
uuid: "7484f8dfdf4f4185ace4afd7fab343f2"
format_stamp: "Formatted at 2026-06-19 15:08:35 on dist-test-slave-zpbw"
server_key: "42a85ca2d9f730d462ff762db349ffba"
server_key_iv: "a925c02f2440ec8c2b69436897f8298d"
server_key_version: "encryptionkey@0"
I20260619 15:08:35.875044 15485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:08:35.894259 15485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:08:35.898352 15485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:08:35.898654 15485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:08:35.917749 15485 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.124:36067
I20260619 15:08:35.917768 15551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.124:36067 every 8 connection(s)
I20260619 15:08:35.919281 15485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/data/info.pb
I20260619 15:08:35.923111 15552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:08:35.924090 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn730WE/build/asan/bin/kudu as pid 15485
I20260619 15:08:35.924304 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn730WE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881351219607-24989-0/minicluster-data/master-2/wal/instance
I20260619 15:08:35.924890 24989 external_mini_cluster.cc:1428] Setting key 68827688f3dd1afe48d55c079963d590
I20260619 15:08:35.932466 15552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:08:35 dist-test-slave-zpbw krb5kdc[15323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
Jun 19 15:08:35 dist-test-slave-zpbw krb5kdc[15323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881715, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:08:35.948247 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.926882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.24.103.124:36067)
0619 15:08:35.927674 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:35.927686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:35.927730 (+ 44us) server_negotiation.cc:407] Connection header received
0619 15:08:35.929888 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:35.929911 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:35.930369 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:35.930790 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:35.932317 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.933732 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.935285 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.936609 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.936814 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.940880 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:35.940889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:35.940900 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:35.940936 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:35.944134 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.944864 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.944870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.944877 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.945017 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.945425 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.945436 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.945438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.945968 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:35.946283 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:35.946740 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 15:08:35.947190 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":371,"threads_started":1}
I20260619 15:08:35.949975 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.935102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35851 (local address 127.0.0.1:50152)
0619 15:08:35.936000 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.936045 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:35.936213 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:35.936944 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:35.936950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:35.937585 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:35.937836 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:35.937845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.938949 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.938953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:35.940017 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:35.940032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.940830 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.940837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:35.941012 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.941705 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:35.941746 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:35.944209 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:35.947410 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:35.947422 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:35.947429 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:35.947865 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:35.948260 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:35.948265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:35.948269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:35.948433 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:35.948969 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:35.948978 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:35.949164 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 15:08:35.949354 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 15:08:35.949691 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":487,"threads_started":1}
I20260619 15:08:35.950057 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.935184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.24.103.126:35851)
0619 15:08:35.936604 (+ 1420us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:35.936609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:35.936624 (+ 15us) server_negotiation.cc:407] Connection header received
0619 15:08:35.936674 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:35.936677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:35.936743 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:35.936873 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:35.938006 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.938726 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.940190 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.940592 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.940760 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.944413 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:35.944418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:35.944421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:35.944455 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:35.947288 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.947995 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.948001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.948004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.948142 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.948560 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.948565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.948569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.948845 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:35.948959 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:35.949443 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 15:08:35.949656 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1320,"thread_start_us":1201,"threads_started":1}
I20260619 15:08:35.951670 15552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:08:35 dist-test-slave-zpbw krb5kdc[15323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881715, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
Jun 19 15:08:35 dist-test-slave-zpbw krb5kdc[15323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260619 15:08:35.964761 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.952279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34055 (local address 127.0.0.1:60166)
0619 15:08:35.952654 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.952672 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:35.952768 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:35.953058 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:35.953061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:35.953431 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:35.953570 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:35.953578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.954649 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.954657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:35.955621 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:35.955631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.956320 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.956325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:35.956501 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.957093 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:35.957110 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:35.959277 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:35.962311 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:35.962317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:35.962321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:35.962703 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:35.963128 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:35.963134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:35.963138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:35.963281 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:35.963801 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:35.963805 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:35.963963 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 15:08:35.964099 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 15:08:35.964389 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260619 15:08:35.964794 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.952460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.24.103.125:34055)
0619 15:08:35.952657 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:35.952663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:35.952710 (+ 47us) server_negotiation.cc:407] Connection header received
0619 15:08:35.952875 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:35.952879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:35.952939 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:35.953041 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:35.953813 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.954509 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.955756 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.956176 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.956311 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.959464 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:35.959470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:35.959477 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:35.959516 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:35.962173 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.962866 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.962872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.962875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.963009 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.963398 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.963407 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.963409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.963672 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:35.963824 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:35.964209 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 15:08:35.964423 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 15:08:35.965694 15552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:08:35.966563 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.951813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50164 (local address 127.24.103.126:35851)
0619 15:08:35.952339 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:35.952344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:35.952360 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:08:35.952409 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:35.952412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:35.952475 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:35.952579 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:35.953939 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.954640 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.955876 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.956286 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.956422 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.960906 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:35.960914 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:35.960921 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:35.960964 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:35.963728 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.964421 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.964434 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.964436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.964557 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.965037 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.965045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.965050 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.965389 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:35.965690 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:35.966005 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 15:08:35.966258 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399}
Jun 19 15:08:35 dist-test-slave-zpbw krb5kdc[15323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881715, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
Jun 19 15:08:35 dist-test-slave-zpbw krb5kdc[15323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260619 15:08:35.980197 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.966555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:55010)
0619 15:08:35.966732 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:35.966751 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:35.966865 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:35.967260 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:35.967264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:35.967717 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:35.967895 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:35.967906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.969230 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.969234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:35.970170 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:35.970195 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.970896 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.970902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:35.971149 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.971773 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:35.971790 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:35.974180 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:35.977363 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:35.977371 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:35.977375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:35.977824 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:35.978199 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:35.978207 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:35.978211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:35.978364 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:35.979005 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:35.979011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:35.979182 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 15:08:35.979379 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 15:08:35.979701 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 15:08:35.980176 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.966834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55010 (local address 127.24.103.124:36067)
0619 15:08:35.966981 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:35.966987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:35.967002 (+ 15us) server_negotiation.cc:407] Connection header received
0619 15:08:35.967046 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:35.967050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:35.967120 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:35.967221 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:35.968044 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.969099 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.970330 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.970720 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.970882 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.974363 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:35.974368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:35.974370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:35.974397 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:35.977234 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.977951 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.977955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.977957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.978069 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.978505 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.978513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.978518 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.978890 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:35.979038 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:35.979499 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 15:08:35.979858 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":50}
I20260619 15:08:35.983807 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:35.968435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.24.103.125:34055)
0619 15:08:35.968600 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:35.968605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:35.968621 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:08:35.968703 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:35.968706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:35.968757 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:35.968829 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:35.970131 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.971069 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.972241 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:35.972937 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:35.973062 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:35.977673 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:35.977681 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:35.977689 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:35.977732 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:35.981067 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.981829 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.981836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.981840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.981986 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:35.982391 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:35.982400 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:35.982402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:35.982680 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:35.982981 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:35.983261 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 15:08:35.983483 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 15:08:35.985759 15552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2: Bootstrap starting.
I20260619 15:08:35.989878 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:08:35.990104 15552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:35.991317 15552 log.cc:824] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:35.993959 15552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2: No bootstrap required, opened a new log
I20260619 15:08:35.992442 15378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "33e6636beff74721aeaec12eca25d3f0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50164
I20260619 15:08:36.001276 15552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } }
I20260619 15:08:36.001662 15552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7484f8dfdf4f4185ace4afd7fab343f2, State: Initialized, Role: FOLLOWER
I20260619 15:08:36.002137 15552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } }
I20260619 15:08:36.003192 15559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } } }
I20260619 15:08:36.003566 15403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } attempt: 4
I20260619 15:08:36.003970 15559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:36.004281 15552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 15:08:36 dist-test-slave-zpbw krb5kdc[15323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881714, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
W20260619 15:08:36.021544 15570 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:08:36.021696 15570 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:08:36.022689 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:36.004249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:55020)
0619 15:08:36.004440 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:36.004461 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:36.004586 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:36.004995 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:36.005000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:36.005529 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:36.005756 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:36.005770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.007582 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.007589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:36.009135 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:36.009154 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.010412 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.010420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:36.010644 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:36.011360 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:36.011383 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:36.014704 (+ 3321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:36.019475 (+ 4771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:36.019482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:36.019487 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:36.020065 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:36.020626 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:36.020634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:36.020638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:36.020814 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:36.021624 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:36.021633 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:36.021805 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 15:08:36.021948 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 15:08:36.022265 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 15:08:36.024009 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:36.004315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.24.103.124:36067)
0619 15:08:36.004567 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.004573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.004598 (+ 25us) server_negotiation.cc:407] Connection header received
0619 15:08:36.004726 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.004733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.004834 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.004957 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.005939 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.007416 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.009299 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.010242 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.010411 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:36.014973 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:36.014982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:36.014989 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:36.015035 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:36.019297 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:36.020247 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:36.020255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:36.020261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:36.020458 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:36.020963 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:36.020968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:36.020972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:36.021445 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:36.023175 (+ 1730us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:36.023320 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 15:08:36.023569 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":113}
I20260619 15:08:36.018939 15485 master_runner.cc:186] Error getting master registration for 127.24.103.126:35851: OK, instance_id { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" instance_seqno: 1781881714854500 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 15:08:36.029917 15485 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 15:08:36.030227 15403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0: Bootstrap starting.
I20260619 15:08:36.033469 15403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:36.034485 15403 log.cc:824] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:36.037402 15403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0: No bootstrap required, opened a new log
I20260619 15:08:36.042853 15403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } }
I20260619 15:08:36.043220 15403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e6636beff74721aeaec12eca25d3f0, State: Initialized, Role: FOLLOWER
I20260619 15:08:36.043711 15403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } }
I20260619 15:08:36.044721 15573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } } }
I20260619 15:08:36.045111 15573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:36.045877 15403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:36.046020 15378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 15:08:36.046257 15378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:36.049800 15378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } }
I20260619 15:08:36.052474 15378 leader_election.cc:288] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [CANDIDATE]: Term 1 election: Requested vote from peers 25fcf70d700f4a13865db2376045beb0 (127.24.103.125:34055), 7484f8dfdf4f4185ace4afd7fab343f2 (127.24.103.124:36067)
I20260619 15:08:36.052656 15527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33e6636beff74721aeaec12eca25d3f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7484f8dfdf4f4185ace4afd7fab343f2"
I20260619 15:08:36.053192 15527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:36.056947 15527 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33e6636beff74721aeaec12eca25d3f0 in term 1.
I20260619 15:08:36.057627 15353 leader_election.cc:302] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33e6636beff74721aeaec12eca25d3f0, 7484f8dfdf4f4185ace4afd7fab343f2; no voters:
I20260619 15:08:36.058173 15573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:08:36.070753 15573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [term 1 LEADER]: Becoming Leader. State: Replica: 33e6636beff74721aeaec12eca25d3f0, State: Running, Role: LEADER
I20260619 15:08:36.071514 15573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } }
I20260619 15:08:36.076427 15582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e6636beff74721aeaec12eca25d3f0. Latest consensus state: current_term: 1 leader_uuid: "33e6636beff74721aeaec12eca25d3f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } } }
I20260619 15:08:36.076954 15582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:36.087569 15588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260619 15:08:36.093660 15357 tablet.cc:2369] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:08:36.098336 15588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:08:36.121505 15477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } attempt: 5
I20260619 15:08:36.122876 15527 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [term 1 FOLLOWER]: Refusing update from remote peer 33e6636beff74721aeaec12eca25d3f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 15:08:36.150902 15506 tablet.cc:2369] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:08:36.150506 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:36.087388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55028 (local address 127.24.103.124:36067)
0619 15:08:36.087583 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.087590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.094555 (+ 6965us) server_negotiation.cc:407] Connection header received
0619 15:08:36.097245 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.097251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.097358 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.097471 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.106581 (+ 9110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.107676 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.117672 (+ 9996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.118426 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.118539 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:36.137155 (+ 18616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:36.137164 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:36.137169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:36.137220 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:36.141555 (+ 4335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:36.145286 (+ 3731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:36.145293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:36.145297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:36.145491 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:36.147131 (+ 1640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:36.147138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:36.147143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:36.147503 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:36.147769 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:36.149835 (+ 2066us) server_negotiation.cc:299] Negotiation successful
0619 15:08:36.150122 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 15:08:36.153805 15582 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:08:36.156054 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:36.087958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.24.103.126:35851)
0619 15:08:36.088136 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.088142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.095194 (+ 7052us) server_negotiation.cc:407] Connection header received
0619 15:08:36.097885 (+ 2691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.097891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.097982 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.098063 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.103186 (+ 5123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.104370 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.115444 (+ 11074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.116133 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.116267 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:36.131373 (+ 15106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:36.131381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:36.131388 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:36.131446 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:36.136519 (+ 5073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:36.144645 (+ 8126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:36.144651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:36.144655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:36.144925 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:36.146148 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:36.146154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:36.146157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:36.146986 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:36.147640 (+ 654us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:36.155399 (+ 7759us) server_negotiation.cc:299] Negotiation successful
0619 15:08:36.155677 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 15:08:36.157158 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:36.095807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.24.103.125:34055)
0619 15:08:36.096035 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.096043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.107445 (+ 11402us) server_negotiation.cc:407] Connection header received
0619 15:08:36.107729 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.107738 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.107835 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.107941 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.110203 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.111373 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.114389 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.115113 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.115225 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:36.137040 (+ 21815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:36.137052 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:36.137060 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:36.137140 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:36.141448 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:36.144093 (+ 2645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:36.144102 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:36.144106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:36.144377 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:36.147914 (+ 3537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:36.147924 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:36.147929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:36.148795 (+ 866us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:36.149287 (+ 492us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:36.156489 (+ 7202us) server_negotiation.cc:299] Negotiation successful
0619 15:08:36.156763 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 15:08:36 dist-test-slave-zpbw krb5kdc[15323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881715, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260619 15:08:36.156800 15336 master_runner.cc:186] Error getting master registration for 127.24.103.125:34055: OK, instance_id { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" instance_seqno: 1781881715387536 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 15:08:36.169473 15336 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 15:08:36.173542 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:36.122099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36067 (local address 127.0.0.1:55036)
0619 15:08:36.122454 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 15:08:36.122473 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 15:08:36.122581 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:08:36.152945 (+ 30364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:08:36.152951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:08:36.153574 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:08:36.153828 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:36.153846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.157778 (+ 3932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.157785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:36.160514 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:08:36.160532 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.161673 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.161682 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:08:36.161918 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:36.162671 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:08:36.162696 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:08:36.165962 (+ 3266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:08:36.170587 (+ 4625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:36.170595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:36.170614 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:36.171174 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:36.171680 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:08:36.171685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:08:36.171688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:08:36.171895 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:08:36.172563 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:08:36.172567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:08:36.172747 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 15:08:36.172894 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 15:08:36.173192 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260619 15:08:36.175252 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:36.122846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.24.103.124:36067)
0619 15:08:36.152536 (+ 29690us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.152543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.152561 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:08:36.152627 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.152632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.152717 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.155768 (+ 3051us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.155923 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.157583 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.160739 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:36.161440 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:36.166184 (+ 4744us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:36.166423 (+ 239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:08:36.166429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:08:36.166434 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:08:36.166468 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:08:36.170388 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:36.171338 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:36.171343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:36.171348 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:36.171520 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:08:36.172045 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:08:36.172049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:08:36.172052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:08:36.172409 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:08:36.174359 (+ 1950us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:36.174499 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 15:08:36.174834 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29532,"thread_start_us":30241,"threads_started":1}
I20260619 15:08:36.187885 15477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0: Bootstrap starting.
I20260619 15:08:36.193281 15559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e6636beff74721aeaec12eca25d3f0. Latest consensus state: current_term: 1 leader_uuid: "33e6636beff74721aeaec12eca25d3f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } } }
I20260619 15:08:36.193629 15559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:36.196682 15595 mvcc.cc:205] Tried to move back new op lower bound from 7298587509199024128 to 7298587509035376640. Current Snapshot: MvccSnapshot[applied={T|T < 7298587509199024128}]
I20260619 15:08:36.200168 15598 mvcc.cc:205] Tried to move back new op lower bound from 7298587509199024128 to 7298587509035376640. Current Snapshot: MvccSnapshot[applied={T|T < 7298587509199024128}]
I20260619 15:08:36.206172 15559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33e6636beff74721aeaec12eca25d3f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } } }
I20260619 15:08:36.206485 15559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:36.208844 15477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0: Neither blocks nor log segments found. Creating new log.
I20260619 15:08:36.213958 15588 catalog_manager.cc:1347] Generated new cluster ID: 4607db1b280140378a86fa7d449adf3c
I20260619 15:08:36.214090 15588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:08:36.213878 15582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33e6636beff74721aeaec12eca25d3f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } } }
I20260619 15:08:36.214221 15582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:36.215329 15477 log.cc:824] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0: Log is configured to *not* fsync() on all Append() calls
I20260619 15:08:36.225921 15477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0: No bootstrap required, opened a new log
I20260619 15:08:36.241757 15588 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:08:36.245199 15588 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:08:36.252856 15477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } }
I20260619 15:08:36.263015 15477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25fcf70d700f4a13865db2376045beb0, State: Initialized, Role: FOLLOWER
I20260619 15:08:36.263746 15477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } }
I20260619 15:08:36.265240 15603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } } }
I20260619 15:08:36.266223 15603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:36.268647 15477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:08:36.270416 15451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:08:36.275077 15451 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [term 1 FOLLOWER]: Refusing update from remote peer 33e6636beff74721aeaec12eca25d3f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:08:36.270601 15452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33e6636beff74721aeaec12eca25d3f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "25fcf70d700f4a13865db2376045beb0"
I20260619 15:08:36.276091 15588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0: Generated new TSK 0
I20260619 15:08:36.277283 15588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:08:36.279222 15452 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33e6636beff74721aeaec12eca25d3f0 in term 1.
I20260619 15:08:36.280443 15573 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:08:36.339305 15603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e6636beff74721aeaec12eca25d3f0. Latest consensus state: current_term: 1 leader_uuid: "33e6636beff74721aeaec12eca25d3f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } } }
I20260619 15:08:36.339277 15609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33e6636beff74721aeaec12eca25d3f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } } }
I20260619 15:08:36.339696 15603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:36.339720 15609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:08:36.383126 15623 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 15:08:36.388514 15557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:36.386727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.24.103.126:35851)
0619 15:08:36.386967 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.386972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.386989 (+ 17us) server_negotiation.cc:407] Connection header received
0619 15:08:36.387046 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.387052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.387139 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.387237 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.388234 (+ 997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50176: BlockingRecv error: recv got EOF from 127.0.0.1:50176 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 15:08:36.389933 15482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:36.387913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.24.103.125:34055)
0619 15:08:36.388101 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.388107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.388335 (+ 228us) server_negotiation.cc:407] Connection header received
0619 15:08:36.388566 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.388586 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.388700 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.388772 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.389725 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60186: BlockingRecv error: recv got EOF from 127.0.0.1:60186 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 15:08:36.397624 15555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:36.394954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.24.103.124:36067)
0619 15:08:36.395164 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.395169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.395791 (+ 622us) server_negotiation.cc:407] Connection header received
0619 15:08:36.396217 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.396223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.396302 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.396407 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.397395 (+ 988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55044: BlockingRecv error: recv got EOF from 127.0.0.1:55044 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 15:08:36.433254 15482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:36.428713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.24.103.125:34055)
0619 15:08:36.428918 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.428925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.431446 (+ 2521us) server_negotiation.cc:407] Connection header received
0619 15:08:36.431710 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.431717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.431871 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.432003 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.432963 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60192: BlockingRecv error: recv got EOF from 127.0.0.1:60192 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 15:08:36.444041 15482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:36.438462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.24.103.125:34055)
0619 15:08:36.438659 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.438667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.442348 (+ 3681us) server_negotiation.cc:407] Connection header received
0619 15:08:36.442679 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.442685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.442811 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.442930 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.443808 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60200: BlockingRecv error: recv got EOF from 127.0.0.1:60200 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 15:08:36.453051 15482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:36.449415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.24.103.125:34055)
0619 15:08:36.449601 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.449608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.451394 (+ 1786us) server_negotiation.cc:407] Connection header received
0619 15:08:36.451621 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.451627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.451750 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.451858 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.452811 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59186: BlockingRecv error: recv got EOF from 127.0.0.1:59186 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 15:08:36.467342 15482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:36.464955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.24.103.125:34055)
0619 15:08:36.465194 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.465200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.465768 (+ 568us) server_negotiation.cc:407] Connection header received
0619 15:08:36.465953 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.465959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.466060 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.466180 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.467063 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59190: BlockingRecv error: recv got EOF from 127.0.0.1:59190 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
W20260619 15:08:36.481557 15482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:36.479154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59200 (local address 127.24.103.125:34055)
0619 15:08:36.479346 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.479353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.479948 (+ 595us) server_negotiation.cc:407] Connection header received
0619 15:08:36.480166 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.480172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.480268 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.480380 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.481310 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59200: BlockingRecv error: recv got EOF from 127.0.0.1:59200 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 15:08:36.505249 15482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:36.502730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.24.103.125:34055)
0619 15:08:36.502973 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.502980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.503542 (+ 562us) server_negotiation.cc:407] Connection header received
0619 15:08:36.503779 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.503785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.503893 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.504004 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.504970 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59202: BlockingRecv error: recv got EOF from 127.0.0.1:59202 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260619 15:08:36.546067 15482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:36.543446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.24.103.125:34055)
0619 15:08:36.543666 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.543673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.544270 (+ 597us) server_negotiation.cc:407] Connection header received
0619 15:08:36.544556 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.544563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.544665 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.544782 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.545794 (+ 1012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59216: BlockingRecv error: recv got EOF from 127.0.0.1:59216 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260619 15:08:36.627599 15482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:36.624901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59228 (local address 127.24.103.125:34055)
0619 15:08:36.625120 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.625128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.625747 (+ 619us) server_negotiation.cc:407] Connection header received
0619 15:08:36.626000 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.626008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.626150 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.626280 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.627367 (+ 1087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59228: BlockingRecv error: recv got EOF from 127.0.0.1:59228 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 15:08:36.761938 15482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:36.759796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.24.103.125:34055)
0619 15:08:36.760052 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:36.760060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:36.760387 (+ 327us) server_negotiation.cc:407] Connection header received
0619 15:08:36.760629 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:36.760637 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:36.760733 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:36.760881 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:36.761671 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59238: BlockingRecv error: recv got EOF from 127.0.0.1:59238 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 15:08:36.797348 15573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33e6636beff74721aeaec12eca25d3f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e6636beff74721aeaec12eca25d3f0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35851 } } peers { permanent_uuid: "25fcf70d700f4a13865db2376045beb0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34055 } } peers { permanent_uuid: "7484f8dfdf4f4185ace4afd7fab343f2" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36067 } } }
I20260619 15:08:36.797674 15573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e6636beff74721aeaec12eca25d3f0 [sys.catalog]: This master's current role is: LEADER
I20260619 15:08:37.024251 15570 catalog_manager.cc:1259] Loaded cluster ID: 4607db1b280140378a86fa7d449adf3c
W20260619 15:08:37.024334 15482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:08:37.022377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.24.103.125:34055)
0619 15:08:37.022589 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:37.022596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:37.022871 (+ 275us) server_negotiation.cc:407] Connection header received
0619 15:08:37.023122 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:37.023128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:37.023226 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:37.023350 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:08:37.024106 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59240: BlockingRecv error: recv got EOF from 127.0.0.1:59240 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 15:08:37.024361 15570 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2: loading cluster ID for follower catalog manager: success
I20260619 15:08:37.027472 15570 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2: acquiring CA information for follower catalog manager: success
I20260619 15:08:37.029549 15570 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7484f8dfdf4f4185ace4afd7fab343f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 15:08:37.387709 15623 catalog_manager.cc:1259] Loaded cluster ID: 4607db1b280140378a86fa7d449adf3c
I20260619 15:08:37.387828 15623 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0: loading cluster ID for follower catalog manager: success
I20260619 15:08:37.390501 15623 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0: acquiring CA information for follower catalog manager: success
I20260619 15:08:37.392544 15623 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25fcf70d700f4a13865db2376045beb0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 15:08:37.550668 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:37.544099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.24.103.125:34055)
0619 15:08:37.545030 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:37.545034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:37.545053 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:08:37.545115 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:37.545119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:37.545276 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:37.545431 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:37.545918 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:37.546830 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:37.548264 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:37.548720 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:37.548871 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:37.549406 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:37.549966 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:37.550143 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:37.550255 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 15:08:37.550344 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":535,"threads_started":1}
I20260619 15:08:37.568794 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:37.557855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.24.103.124:36067)
0619 15:08:37.563157 (+ 5302us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:37.563161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:37.563181 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:08:37.563283 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:37.563286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:37.563456 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:37.563660 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:37.564220 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:37.564963 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:37.566376 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:37.566874 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:37.567016 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:37.567555 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:37.568107 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:37.568267 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:37.568376 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 15:08:37.568464 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5180,"thread_start_us":5081,"threads_started":1}
I20260619 15:08:37.596793 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:37.575037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.24.103.126:35851)
0619 15:08:37.588000 (+ 12963us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:37.588008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:37.588037 (+ 29us) server_negotiation.cc:407] Connection header received
0619 15:08:37.588405 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:37.588411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:37.588660 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:37.588844 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:37.589708 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:37.591255 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:37.593754 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:37.594389 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:37.594513 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:37.595279 (+ 766us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:37.595775 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:37.595884 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:37.596355 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 15:08:37.596446 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12814,"thread_start_us":13005,"threads_started":1}
I20260619 15:08:37.597064 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:37.575674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.24.103.124:36067)
0619 15:08:37.575866 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:37.575872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:37.588776 (+ 12904us) server_negotiation.cc:407] Connection header received
0619 15:08:37.589093 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:37.589100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:37.589211 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:37.589280 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:37.590129 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:37.591288 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:37.594631 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:37.595356 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:37.595462 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:37.596222 (+ 760us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:37.596357 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:37.596421 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:37.596684 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 15:08:37.596779 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 15:08:37.599355 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:08:37.588851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.24.103.125:34055)
0619 15:08:37.589047 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 15:08:37.589053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:08:37.592445 (+ 3392us) server_negotiation.cc:407] Connection header received
0619 15:08:37.592619 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:08:37.592625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:08:37.592723 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:08:37.592830 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:08:37.593755 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:37.594983 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:37.597082 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:08:37.597678 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:08:37.597784 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:08:37.598466 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:08:37.598595 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:08:37.598661 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 15:08:37.598921 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 15:08:37.599035 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 15:08:37.602283 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 15336
I20260619 15:08:37.604100 15398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:08:37.719306 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 15336
I20260619 15:08:37.740880 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 15410
W20260619 15:08:37.740940 15428 connection.cc:535] server connection from 127.0.0.1:60156 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 15:08:37.742306 15472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:08:37.857920 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 15410
I20260619 15:08:37.875346 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskn730WE/build/asan/bin/kudu pid 15485
I20260619 15:08:37.876509 15547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 15:08:37.984339 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn730WE/build/asan/bin/kudu with pid 15485
2026-06-19T15:08:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4983 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9517 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (366731 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST